<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>Development of EPM Systems: Management Principles and Tools</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Dmitry Is</string-name>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>National Research University Higher School of Economics</institution>
          ,
          <addr-line>20, M yasnitskaya Street, M oscow 101000</addr-line>
          ,
          <country country="RU">Russia</country>
        </aff>
      </contrib-group>
      <abstract>
        <p>This paper focuses on management issues related with implementation and development of enterprise performance management (EPM ) systems. Such systems are used for certain management tasks - strategic analysis, management by objectives, corporate planning and budgeting, corporate reporting. Implementation and development of EPM systems should rely on appropriate methodological approach and supporting technologies. However, at present a holistic approach to EPM systems development management is unavailable. The aim of the research is to determine peculiarities of EPM systems development, basic management principles and management tools applicable on different stages of the development process. Here the management principles proposed are system approach (considering the system as a set of structural components and interrelationships between them), going concern (assumption that an organization intends to continue its activities in the foreseeable future), program-based approach (considering an EPM system development program as a set of interrelated projects), competitive consideration (forming few alternative development programs and selecting one of them for implementation), simulation (modeling of development programs and their consequences taking into account stochastic factors), comprehensive assessment (comparison of alternative programs relying on both quantitative measures and qualitative factors), value for money (comparison of investments with the system's maturity represented by non-financial indicators), aggregated resources (considering resources in aggregate, as total amounts of related payments) and monitoring (analysis of the system's changes over time, both historical and forecasted). Appropriate management tools comprise conceptual modeling, maturity assessment, discreteevent simulation and multi-criteria decision making. These management principles and tools are essential for creating comprehensive methodology of EPM systems development.</p>
      </abstract>
      <kwd-group>
        <kwd>Performance M anagement</kwd>
        <kwd>EPM System</kwd>
        <kwd>Development Program</kwd>
        <kwd>M anagement Principle</kwd>
        <kwd>M anagement Tool</kwd>
      </kwd-group>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>-</title>
      <p>These days many companies face with certain problems related with information
support of corporate governance and strategic management . Such problems include
inconsistency of strategic, tactical and operational management levels (so called
“strategic gap”), lack of business agility, shortcomings in timeliness and accuracy of
corporate reporting. Enterprise performance management (EPM) systems allow
managers to solve or, at least, mitigate such problems. An EPM system is an instrument for
management, but it is also a management object: it should be implemented, monitored
and developed. Therefore, it is a question regarding a comprehensive methodology of
EPM systems development management. Such methodology should rely on certain
principles that, in turn, are derived from key features of EPM systems.
2</p>
    </sec>
    <sec id="sec-2">
      <title>Nature and Key Features of EPM Systems</title>
      <p>
        There are different definitions of EPM systems. For example, Ferreira and Otley
define performance management in wide sense, as “evolving formal and informal
mechanisms, processes, systems, and networks used by organizations for conveying the
key objectives and goals elicited by management, for assisting the strategic process
and ongoing management through analysis, planning, measurement, control,
rewarding, and broadly managing performance, and for supporting and facilitating
organizational learning and change” [
        <xref ref-type="bibr" rid="ref1">1</xref>
        ]. Other (but similar) definitions are advanced by
Coveney et al. [
        <xref ref-type="bibr" rid="ref2">2</xref>
        ], Cokins [
        <xref ref-type="bibr" rid="ref3">3</xref>
        ], Broadbent and Laughlin [
        <xref ref-type="bibr" rid="ref4">4</xref>
        ], and some others.
Appropriate software is developed by leading IT vendors – Oracle, IBM, SAP, SAS and
some other companies. Despite certain differences in definitions, few important
characteristics of EPM systems may be highlighted.
      </p>
      <p>One of the features is complexity and modular structure of an EPM system. The
system comprise special methods, models, information systems, management
processes, as well as management personnel involved into these processes. Regarding
management functions, EPM covers strategic analysis, management by objectives,
corporate planning and budgeting, and corporate reporting (including preparing
consolidated financial statements).</p>
      <p>EPM systems have large-scale scope: they cover entire organizations, large
business segments or groups of companies. As a result, management personnel involved
into the performance management processes may be located in different geographical
points.</p>
      <p>Regarding time span, EPM systems consider long horizons of planning, accounting
and analysis. This distinguishes them from well-known enterprise performance
management (ERP) systems, which deal with individual operations.</p>
      <p>From the point of view of data processing, EPM systems deal with aggregated
management information, which may be both financial and non-financial. This again
distinguishes them from ERP systems , that process detailed operational data.</p>
      <p>The foregoing features seem essential for further formulation of peculiarities and
principles of EPM systems development management.</p>
    </sec>
    <sec id="sec-3">
      <title>Existing Developments Related to EPM Systems</title>
      <p>There are four groups of works associated with EPM systems, their implementation,
maintenance and development. All of them are valuable in some aspects, but also
have certain limitations.</p>
      <p>The first group includes works describing nature and scope of EPM systems (some
of such books and articles are mentioned above). These publications answer questions
about the concept of performance management and its role in modern business. The
importance of the theoretical developments is that allow to define main functional
features of EPM systems and their linkage with real management tasks. However,
these works do not examine methods and tools for creating and developing EPM
systems.</p>
      <p>
        The second group of existing developments deals with requirements to EPM
systems. The examples are different codes of corporate governance [
        <xref ref-type="bibr" rid="ref5">5</xref>
        ] and studies in the
field of strategic management [
        <xref ref-type="bibr" rid="ref6">6</xref>
        ]. Establishing goals of EPM systems these
developments, again, do not consider implementing and developing matters.
      </p>
      <p>
        The third group is associated with methods, models and technologies that make up
internal content of EPM systems. In this regard, there are works in the fields of
management accounting [
        <xref ref-type="bibr" rid="ref7">7</xref>
        ], scorecarding [
        <xref ref-type="bibr" rid="ref8 ref9">8, 9</xref>
        ], planning and budgeting [
        <xref ref-type="bibr" rid="ref10">10</xref>
        ], financial
consolidation and business analysis [
        <xref ref-type="bibr" rid="ref11">11</xref>
        ]. All this is important for understanding
components of EPM systems and their peculiarities for organizations of different types
and industries. However, the questions regarding integration of the components and
consistency of their implementation are not discussed.
      </p>
      <p>
        Finally, existing works of the forth group deal with different aspects of information
systems development management: decision making in the IT sphere [
        <xref ref-type="bibr" rid="ref12">12</xref>
        ], IT
scorecarding [
        <xref ref-type="bibr" rid="ref13">13</xref>
        ], IT project management [
        <xref ref-type="bibr" rid="ref14 ref15">14, 15</xref>
        ] and enterprise architecture [
        <xref ref-type="bibr" rid="ref16">16</xref>
        ]. At the
same time, such particular aspects are not completely integrated and therefore do not
form a holistic methodology. Furthermore, they have general nature and do not
consider key features of EPM systems , which distinguish them from other classes of
management systems.
      </p>
      <p>Therefore, a comprehensive methodological approach to EPM systems
development should rely on existing developments of all four groups , which in aggregate
define the nature of such systems, their purposes, main components and some aspect s
of their development. At the same time, the proposed methodological approach has
two important distinctive features : its integrity (considering relationships between key
management aspects) and focus on EPM peculiarity (taking into account distinctive
features of such systems). These two points form the basis for formulating basic
principles of EPM systems development management.
4</p>
    </sec>
    <sec id="sec-4">
      <title>Principles of EPM Systems Development Management</title>
      <p>There are three basic peculiarities of EPM systems development management. First,
economic benefits from implementing and developing such systems are implicit,
because improved management information has indirect effect on financial position of
organizations. Second, there are factors of uncertainty related with development
projects and their impact on the EPM system. Third, some of the development projects
are complicated: there are projects with uncertain outcomes, iterative project s (with
possibility of re-executing) and projects with multiple variants of implementation.
Taking into account features of EPM systems and peculiarities of their development
the following management principles may be formulated.</p>
      <p>The system approach principle means considering EPM system as a set of
structural components and interrelationships between them.</p>
      <p>
        The going concern principle is similar to one formulated in the International
Financial Reporting Standards [
        <xref ref-type="bibr" rid="ref17">17</xref>
        ]. It means that an organization intends to continue its
activities in the foreseeable future.
      </p>
      <p>The program-based approach principle means that EPM system developing is
realized through certain projects, which in aggregate form a development program.</p>
      <p>The competitive consideration principle implies forming potential (alternative)
development programs based on business needs and financing limits. Then the most
preferable of these programs is to be s elected for implementation.</p>
      <p>The simulation principle is related with discrete-event simulation of potential
development programs and their consequences. The results of such modeling is the basis
for evaluating alternative programs and selecting the most preferable one.</p>
      <p>The principle of comprehensive assessment of potential development programs
means that their comparison is based not only on quantitative measures (results of
simulation) but also on qualitative factors. In this regard, expert judgments are
reasonable.</p>
      <p>The value for money principle means comparison of investments in EPM system
development with non-financial indicators characterizing the system’s maturity and
its dynamics.</p>
      <p>The principle of aggregated resources means that resources required for an EPM
system development are not considered in details. Instead, they are evaluated in
aggregate, as total amounts of payments related with individual projects. An assumption
that all the resources are unlimited and assessable is also applied.</p>
      <p>The principle of EPM development monitoring refers both to the system’s
development history, and forecasting its state in the future.
5</p>
    </sec>
    <sec id="sec-5">
      <title>Management Tools</title>
      <p>The foregoing principles of EPM systems development explain applicability of few
management tools that can be used on different stages of the management process.
Such tools include info-logical conceptual modeling, formation of EPM maturity
models, discrete-event simulation and multi-criteria decision making. Let us to
consider these tools in more details.</p>
      <sec id="sec-5-1">
        <title>Conceptual Modeling</title>
        <p>
          An info-logical conceptual modeling comes from the system approach principle. A
conceptual model describes an EPM system as a set of functional blocks , functional
modules and information flows between them [
          <xref ref-type="bibr" rid="ref18">18</xref>
          ]. There are four major functional
blocks: strategic analysis and strategic choice, management by objectives, corporate
planning and budgeting, and corporate reporting. Each block contains few functional
modules. For example, the block of strategic analysis and strategic choice includes
modules for monitoring, forecasting, corporate appraisal and p ositioning, goals
setting, global strategic initiatives determining, and high-level objectives setting. The
management by objectives block includes modules for scenarios determination,
targets setting and objectives achievement analysis. The block of corporate planning an d
budgeting includes two modules : plans and budgets preparation and budgetary
control. Finally, the corporate reporting block also includes two modules: corporate
statements preparation and corporate statements analysis.
        </p>
        <p>The model also des cribes certain external (i.e. located outside the EPM system)
objects – external environment, external stakeholders , accounting systems , sources of
qualitative information and operational management systems .</p>
        <p>The information flows connecting functional blocks, functional modules and
external objects include environmental and accounting information, goals and objectives,
strategic initiatives, forecasts and plans, results of objectives achievement analysis
and budgetary control, as well as external and management statements.</p>
        <p>Further developments in the field of conceptual modeling include developing
approaches and procedures for forming individual and industry related models for
organizations of different types and industries.
5.2</p>
      </sec>
      <sec id="sec-5-2">
        <title>EPM Maturity Model</title>
        <p>
          Maturity modeling of EPM systems derives from the principles of system approach,
value for money and development monitoring. The aim of such models is representing
qualitative and quantitative capabilities of EPM systems according to the stages of
their maturity, in terms of threshold levels – from initial to improved [
          <xref ref-type="bibr" rid="ref19">19</xref>
          ].
        </p>
        <p>The maturity model is a base for determining framework for monitoring and
planning EPM systems development. For this purpose, two target maturity levels are
considered: lower (satisfactory) and upper (advanced). The target levels may be
permanent or changing over time, depending on external conditions, new technological
solutions and an organization’s vision. Target maturity levels, in turn, define three
maturity zones: unsatisfactory (below the lower target maturity level), satisfactory
(between the lower and upper levels ) and advanced (above the upper maturity level).</p>
        <p>The trajectory of an EPM system development represents chan ging the system’s
maturity index over time with respect to threshold maturity levels. The trajectories
can be of different types – actual, planned, forecasted or simulated.</p>
        <p>Further developments in this field are associated with methodology of forming the
performance management index for individual organizations, a s well as indicators
characterizing effectiveness of EPM development programs.
5.3</p>
      </sec>
      <sec id="sec-5-3">
        <title>Development Program Simulation</title>
        <p>Another management tool is associated with simulation of EPM development
programs, according to the simulation principle. Simulation seems essential because
many parameters (duration of projects, usage of resources, impact on maturity level)
are stochastic.</p>
        <p>
          First of all, simulation refers to potential alternative programs – to obtain
information about their executing and results, for subsequent selecting a program for
implementation. However, simulation may also be applied for an approved program that
has already started, for forecasting purposes. In any case, we mean simulation of
discrete-event type, which describes behavior of the modeled system as a discrete
sequence of events in time [
          <xref ref-type="bibr" rid="ref20">20</xref>
          ]. This is explained by the nature of development
programs: each of them consists of a set of interrelated projects affecting both EPM
maturity and financial metrics. So, implementation of a program can be presented as a
sequence of discrete events: beginning and ends of projects, as well as impacts on
maturity index and financial metrics.
        </p>
        <p>Simulation seems quite effective for complicated projects – with uncertain
outcomes, possibility of re-executing and multiple options of implementation. This is
explained by the fact that parameters of such projects (for example, specific outcome
or certain way of implementation) are also stochastic.</p>
        <p>
          There are special information systems applicable for discrete -event simulation,
such as Arena [
          <xref ref-type="bibr" rid="ref21 ref22 ref23">21, 22, 23</xref>
          ], AnyLogic [
          <xref ref-type="bibr" rid="ref24">24</xref>
          ] and some others. Details of using such
software, as well as determining key elements of the simulation models and
constructing special indicators of aggregated modeling results may be considered as further
research areas.
5.4
        </p>
      </sec>
      <sec id="sec-5-4">
        <title>Multi-Criteria Decision Making</title>
        <p>
          Multi-criteria decision making deals with a broad range of practical tasks in all areas
of human life, including business, public administration, engineering, politics,
medicine, etc. [
          <xref ref-type="bibr" rid="ref25 ref26">25, 26</xref>
          ]. In our case, decision making is required for comparing alternative
development programs and selecting one of them for implementation. This is relevant
to aforementioned principles of competitive consideration (selecting a program for
implementation) and comprehensive assessment (availability of several assessment
criteria). Additionally, the value for money principle means that criteria of assessment
may be financial and non-financial, quantitative and qualitative.
        </p>
        <p>Financial part of the assessment is related with resources used for EPM
development. According to the principle of aggregated resources, all of them are considered
as cumulative expenditures assigned to the time scale. Financial figures, as well as
maturity related quantitative indicators are obtained from the simulation process.
Additionally semi-structured, informal qualitative factors may be taken into account.
Such factors can not be measured, but may be estimated by experts. Moreover, the
role of experts may be expanded: they may be invited for interpreting some (or all)
quantitative figures.</p>
        <p>Therefore, the decision is made relying on analysis and generalization of few
factors (criteria), which have different nature. Such analysis should rely on simulation
results and experts’ judgements. Consequently, an approach to evaluating the
alternatives and subsequent decision making is essential. This point is also a subject of
further research.
6</p>
      </sec>
    </sec>
    <sec id="sec-6">
      <title>Conclusion</title>
      <p>Enterprise performance management (EPM) systems have certain specific features –
modularity, large-scale scope, long-term planning horizons and using aggregated
management information. These features lead to peculiarities of EPM development –
implicit economic benefits, factors of uncertainty, complicated projects. All this
allows us to formulate principles of EPM development management – system approach,
going concern, program-based approach, simulation and comprehensive assessment
of development programs, value for money, aggregated considering of resources and
monitoring of development. In turn, these principles explain reasonabilit y of applying
such management tools as info-logical conceptual modeling, using maturity models,
simulation of development programs and multi-criteria decision making.</p>
      <p>The defined principles and tools are compliant with existing developments related
with nature and scope of EPM systems, requirements to them, their components and
different aspects of their development management. However, the difference between
the proposed methodological approach and existing developments is explained by its
integrity and taking into account specific features of EPM systems.</p>
      <p>Foregoing features, principles and management tools are closely interrelated.
Taken together, they form reliable basis for detailed study of particular aspects of EPM
systems development management and creating a comprehensive methodological
approach in this field.</p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          1.
          <string-name>
            <surname>Ferreira</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Otley</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          :
          <article-title>The design and use of performance management systems: An extended framework for analysis</article-title>
          .
          <source>M anagement Accounting Research</source>
          <volume>20</volume>
          ,
          <fpage>263</fpage>
          -
          <lpage>282</lpage>
          (
          <year>2009</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          2.
          <string-name>
            <surname>Coveney</surname>
            ,
            <given-names>M .</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Ganster</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Hartlen</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>King</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          :
          <article-title>The strategy gap: Leveraging technology to execute winning strategies</article-title>
          . Wiley, Hoboken, NJ (
          <year>2003</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          3.
          <string-name>
            <surname>Cokins</surname>
          </string-name>
          , G.:
          <article-title>Performance management: Finding the missing pieces (to close the intelligence gap)</article-title>
          . Wiley, Hoboken, NJ (
          <year>2004</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          4.
          <string-name>
            <surname>Broadbent</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Laughlin</surname>
          </string-name>
          , R.:
          <article-title>Performance management systems: A conceptual model</article-title>
          .
          <source>M anagement Accounting Research</source>
          <volume>20</volume>
          ,
          <fpage>283</fpage>
          -
          <lpage>295</lpage>
          (
          <year>2009</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          5.
          <string-name>
            <surname>Tricker</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          :
          <article-title>Corporate governance: Principles, policies, and practices</article-title>
          .
          <source>Third edition</source>
          . Oxford University Press, Oxford, UK (
          <year>2015</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          6. M intzberg, H.,
          <string-name>
            <surname>Ahlstrand</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Lampel</surname>
          </string-name>
          , J.:
          <article-title>Strategy safari: Your complete guide through the wilds of strategic management. Second edition</article-title>
          . Prentice Hall (
          <year>2009</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          7.
          <string-name>
            <surname>Drury</surname>
            ,
            <given-names>C.:</given-names>
          </string-name>
          <article-title>M anagement accounting for business</article-title>
          .
          <source>Cengage Learning</source>
          , UK (
          <year>2016</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          8.
          <string-name>
            <surname>Kaplan</surname>
            ,
            <given-names>R.S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Norton</surname>
            ,
            <given-names>D.P.:</given-names>
          </string-name>
          <article-title>The balanced scorecard: Translating strategy into action</article-title>
          . Harvard Business Review Press (
          <year>1996</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          9.
          <string-name>
            <surname>Niven</surname>
            ,
            <given-names>P.R.</given-names>
          </string-name>
          :
          <article-title>Balanced scorecard evolution: A dynamic approach to strategy execution</article-title>
          . Wiley, Hoboken, NJ (
          <year>2014</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          10.
          <string-name>
            <surname>Bragg</surname>
            ,
            <given-names>S.M .:</given-names>
          </string-name>
          <article-title>Budgeting: The comprehensive guide</article-title>
          .
          <source>Fourth edition. AccountingTools</source>
          , Centennial, CO (
          <year>2017</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          11.
          <article-title>A guide to the business analysis body of knowledge (BABOK Guide)</article-title>
          .
          <source>Third edition. IIBA</source>
          , Toronto (
          <year>2015</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          12.
          <string-name>
            <surname>Schniederjans</surname>
            ,
            <given-names>M .J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Hamaker</surname>
            ,
            <given-names>J.L.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Schniederjans</surname>
            ,
            <given-names>A.M .:</given-names>
          </string-name>
          <article-title>Information technology investment: Decision-making methodology</article-title>
          .
          <source>World Scientific Publishing</source>
          (
          <year>2004</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          13.
          <string-name>
            <surname>Keyes</surname>
          </string-name>
          , J.:
          <article-title>Implementing the IT balanced scorecard: Aligning IT with corp orate strategy</article-title>
          .
          <source>Auerbach Publications</source>
          (
          <year>2005</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          14.
          <string-name>
            <surname>Ward</surname>
          </string-name>
          , J.,
          <string-name>
            <surname>Daniel</surname>
          </string-name>
          , E.:
          <article-title>Benefits management: How to increase the business value of y our IT projects</article-title>
          .
          <article-title>Second edition</article-title>
          . Wiley , UK (
          <year>2012</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          15.
          <string-name>
            <surname>Schwalbe</surname>
            ,
            <given-names>K.</given-names>
          </string-name>
          :
          <article-title>Information technology project management. Eighth edition</article-title>
          .
          <source>Cengage Learning</source>
          , Boston (
          <year>2016</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref16">
        <mixed-citation>
          16.
          <string-name>
            <surname>Kotusev</surname>
            ,
            <given-names>S.:</given-names>
          </string-name>
          <article-title>The practice of enterp rise architecture: A modern approach to business and IT alignment</article-title>
          . SK Publishing, M elbourne (
          <year>2018</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref17">
        <mixed-citation>
          17.
          <article-title>The conceptual framework for financial reporting (Conceptual framework)</article-title>
          .
          <source>IASB</source>
          , London (
          <year>2018</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref18">
        <mixed-citation>
          18.
          <string-name>
            <surname>Isaev</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          :
          <article-title>Performance management information support system: A conceptual model</article-title>
          .
          <source>Europ ean Journal of Economics, Finance and Administrative Sciences</source>
          <volume>52</volume>
          ,
          <fpage>6</fpage>
          -
          <lpage>20</lpage>
          (
          <year>2012</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref19">
        <mixed-citation>
          19.
          <string-name>
            <surname>Aho</surname>
            ,
            <given-names>M .:</given-names>
          </string-name>
          <article-title>A construct for performance management maturity assessment</article-title>
          .
          <source>Doctoral thesis</source>
          . Tampere University of Technology, Tampere, Finland (
          <year>2011</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref20">
        <mixed-citation>
          20.
          <string-name>
            <surname>Choi</surname>
            ,
            <given-names>B.K.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Kang</surname>
            ,
            <given-names>D.:</given-names>
          </string-name>
          <article-title>M odeling and simulation of discrete-event systems</article-title>
          . Wiley, New York (
          <year>2013</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref21">
        <mixed-citation>
          21.
          <string-name>
            <surname>Kelton</surname>
          </string-name>
          , W.D.,
          <string-name>
            <surname>Sadowski</surname>
            ,
            <given-names>R.P.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Swets</surname>
            ,
            <given-names>N.B.</given-names>
          </string-name>
          :
          <article-title>Simulation with Arena. M cGraw-</article-title>
          <string-name>
            <surname>Hill</surname>
          </string-name>
          , London (
          <year>2014</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref22">
        <mixed-citation>
          22.
          <string-name>
            <surname>Rossetti</surname>
          </string-name>
          , M .D.:
          <article-title>Simulation modeling</article-title>
          and Arena. Wiley, New York (
          <year>2015</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref23">
        <mixed-citation>
          23.
          <string-name>
            <surname>Cosgrove</surname>
          </string-name>
          , W.J.:
          <article-title>Simplify ing PERT Network Simulation with Arena</article-title>
          .
          <source>California Journal of Operations M anagement 6(1)</source>
          ,
          <fpage>61</fpage>
          -
          <lpage>68</lpage>
          (
          <year>2008</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref24">
        <mixed-citation>
          24.
          <string-name>
            <surname>Borshchev</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          :
          <article-title>The big book of simulation modeling. M ultimethod modeling with AnyLogic 6</article-title>
          . AnyLogic North America, Oakbrook Terrace, IL (
          <year>2015</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref25">
        <mixed-citation>
          25.
          <string-name>
            <surname>Zopounidis</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Doumpos</surname>
            ,
            <given-names>M .:</given-names>
          </string-name>
          <article-title>M ultiple criteria decision making: Applications in management and engineering</article-title>
          . Springer (
          <year>2017</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref26">
        <mixed-citation>
          26.
          <string-name>
            <surname>Hontoria</surname>
            ,
            <given-names>E.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Saez</surname>
            ,
            <given-names>F.J.,</given-names>
          </string-name>
          <article-title>M unier</article-title>
          , N.:
          <article-title>Strategic approach in multi-criteria decision making: A practical guide for complex scenarios</article-title>
          . Springer (
          <year>2019</year>
          ).
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>