=Paper=
{{Paper
|id=Vol-2753/paper27
|storemode=property
|title=Development of a Markov Model of Changes in Patients' Health Conditions in Medical Projects
|pdfUrl=https://ceur-ws.org/Vol-2753/paper17.pdf
|volume=Vol-2753
|authors=Olga Mezentseva,Oleksii Kolesnikov,Katerina Kolesnikova
|dblpUrl=https://dblp.org/rec/conf/iddm/MezentsevaKK20
}}
==Development of a Markov Model of Changes in Patients' Health Conditions in Medical Projects==
https://ceur-ws.org/Vol-2753/paper17.pdf
Development of a Markov Model of Changes in Patients' Health
Conditions in Medical Projects
Olga Mezentseva,a, Oleksii Kolesnikova and Kateryna Kolesnikovaa
a
Taras Shevchenko National University of Kyiv, 60 Volodymyrska Street, Kyiv, 01033, Ukraine
Abstract
In this article, there is discussed the components of the forecast model by changes in the condition
of patients in the management of medical projects. We proposed the use of Markov chains to
obtain quantitative results and take into account random processes in the management of medical
projects. In recent years, the problem of health care of Ukrainian citizens has become a threat
to national security. The main reason for the current situation lies in the imperfect health care
management system, insufficient funding and irrational distribution of funds allocated by the
state to the industry. Resolving the contradictions between the needs of the population of
Ukraine in timely and quality medical care and the tasks of medical institutions in providing
these services is possible only if the implementation in the medical field of project
management methodology based on the use of mathematical modeling methods. During the
implementation of medical services projects there is a need to assess the prospects for the use
of new forms of services in different environments of the project. This estimate can be
performed using mathematical models. At present, such models are based on the
approximation of real data, which reflect the delay in the effectiveness of medical institutions
with patients. In this case, the projects have already begun - they are implemented, and
experimental models reflect a certain result. However, at the stage of preliminary project
preparation it is necessary to determine the expected results in the "prediction" mode. Such
properties are inherent in Markov models, which are built for specific states of the system,
and can be used for cases of changes in the internal characteristics of the system. In the
course of the research we built a model of realization of Markov processes using the
construction of the process graph of the WHO model and the C6 model of medical project
management.
Keywords 1
Medical services, Markov Model, medical project
1. Introduction
Current trends in project management are aimed at transforming projects into dynamic systems that are not
only subject to market requirements, but also through the use of modern models are continuously improved
on the basis of proactive approaches to change management. Existing health care project management
systems do not always provide solutions to improve quality and accessibility, due to the lack of effective
models, methods, tools for evaluating project results for the implementation of management mechanisms,
including through feedback. Therefore, the development of models that reflect the state of the health care
system and the formation on their basis of mechanisms for proactive management of projects in the field of
health care will ensure the quality and accessibility of health care services.
The introduction of project management in the field of medical services in medical institutions
operating in a turbulent competitive environment, necessitates the management of quality and cost of
treatment of patients in medical projects with continuous improvement of the content and system of health
IDDM’2020: 3rd International Conference on Informatics & Data-Driven Medicine, November 19–21, 2020, Lviv, Ukraine
EMAIL: olga.mezentseva.fit@gmail.com (Olga Mezentseva); akoles78@gmail.com (Oleksii Kolesnikov); amberk4@gmail.com (Kateryna
Kolesnikova)
ORCID: 0000-0002-8430-4022 (Olga Mezentseva); 0000-0003-2366-1920 (Oleksii Kolesnikov); 0000-0002-9160-59823 (Kateryna
Kolesnikova)
©️ 2020 Copyright for this paper by its authors.
Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR Workshop Proceedings (CEUR-WS.org)
care [19]. Under conditions of fierce competition for project-based medical institutions and institutions,
together with the need to improve the mechanisms of product (service), process, development and business
values, the issue of improving project management models and methods that are drivers of innovative
development to expand the range of medical services.
The design of medical services is the most important condition for the successful implementation of
medical activities, which is usually considered the art of the doctor. But the management of medical
services also contains an organizational and technical component - planning, implementation of treatment
projects, control, analysis and correction of results. When managing health care projects, one of the main
tasks assesses the effectiveness of projects. In a general sense, measuring the effectiveness of treatment
projects are expressed in the study of the health of the patient community. Due to the lack of models and
methods for early evaluation of the effectiveness of treatment projects, as a rule, they are planned based on
the results of best practice. Usually the evaluation of efficiency is carried out due to intuitive assumptions
or methods of field observations. But this approach allows you to evaluate existing treatment projects,
which by definition reduces the value of the service. Therefore, for proactive management of projects for
the provision of medical services, the task of early assessment of the expected result is already
relevant when planning.
In a general sense, measuring the effectiveness of treatment projects is expressed in the study of
the health of the patient community.
2. Analysis of recent research and publications
Analysis of world experience in the organization of the health care system has shown the feasibility of
using a project approach that allows you to most effectively solve the problem of achieving this goal in a
limited time, financial, material, human, etc. types of resources. Therefore, it is time to create conditions
for the transformation of medical projects in the direction of proactive management through the use of
models that reflect the essential features of the studied system - the patient community.
Application of Markov processes to random processes in various branches of applied science
swallowed in [1-7, 17-20]. [9] proposes new models of innovation management in projects based on the
use of Markov processes. This approach can be used for further development of medical projects. [10]
describes Markov models with changes that can be applied to medical projects. However, there are no
methods of interaction between groups of patients.
In [11] a detailed description of the technological approach to the management of any project. At the
same time [12] describes integrated methodologies of organizational management and management of
products and services of medical projects. However, methods for evaluating the effectiveness of
communicative interaction are insufficiently described. In [13, 14, 16] approaches to the management of
complex IT projects based on predictive management using artificial intelligence methods, in particular,
using artificial neural networks. However, studies based on the interaction of random variables have not
been conducted.
The goal of the article is to substantiate and develop a model for predicting the effectiveness of medical
projects based on random processes using data analysis of grouping processes in a medical project and the
application of modeling of Markov processes of changes in patients.
3. Model development and use of modelling method
To build a Markov model of changing the health status of the community of consumers of medical
services, it is necessary to decompose the system into specific conditions and build a scheme of transitions
between these conditions. An important aspect of developing a Markov model is the method of
determining the calculation of transition probabilities.
One of the most well-known models for qualitative reflection of system states is the generally
accepted scheme of the World Health Organization (WHO), the concept of which is based on the
obvious fact that there are certain transitions from one state to another. The scheme of reactions of the
population according to WHO materials, is given in fig. 1, reflects the component of public health as a
result of the impact on people of environmental factors. The area of each of the zones is proportional
to the share of the population with the corresponding signs of influence.
5
4
3
2
1
The share of the population with signs of influence
Figure 1: The reaction of the population to the effects of harmful environmental factors:
1 - accumulation of chemicals in human organs and tissues; 2 - physiological and other shifts of
unknown origin; 3 - signs of the disease; 4 - morbidity; 5 - mortality.
The WHO model assumes that in order to go from the 1st to the 2nd state, one must be in the 1st
state. At the same time, there are always more patients of a lower level than those who have moved to
a higher level. All these transitions are calculated from the total number of patients in the market
segment. This model [5] allows you to estimate the approximate volume of health services, and on the
other hand allows you to set the right goals for the management of health care projects. The transition
from each condition will be due to different features in terms of medical institutions. This approach is
not entirely correct, because the transition from the 1st state is possible in a turbulent environment to
any other state. Moreover, as a result of medical measures it is possible to move to a state with a
lower serial number.
The WHO model allows only a qualitative assessment of the effectiveness of various health care
projects and to develop the most effective strategy for promoting a particular medical service on the
market. Many processes in medical activity develop as random processes. The WHO model does not
allow to obtain quantitative results of project effectiveness. The obvious contradiction is the need to
develop a strategy for providing quality medical services not on the basis of modeling these processes,
but based on the method of trial and error in management decisions [8].
To model the change in the state of health of the population, it is proposed to allocate 6 states, in
one of which with a certain probability can be each agent of the system (Fig. 2). Denote by Sі {i=1,
2,…, 6} possible conditions of some community of patients of consumers of medical services caused
by carrying out projects: S1 - practically healthy; S2 - able to work; S3 - temporarily incapacitated; S4 -
chronic disease; S5 - critical condition; S6- exit (death, emigration, etc.).
These conditions of the S1-S6 system form a complete list of health conditions of a certain group of
the population. In the general case, the system of states S1-S6 can be represented by an oriented graph
(Fig. 2).
Figure 2: The oriented graph of the WHO model
Existing transitions between different states are determined by expert assessments. A matrix
containing all the transition probabilities of the Markov chain shown in Fig. 2, has the form:
π1.1 π1.2 π1.3 π1.4 π1.5 π1.6
π 2.1 π 2.2 π 2.3 π 2.4 π 2.5 π 2.6
π 3.1 π 3.2 π 3.3 π 3.4 π 3.5 π 3.6
ij .
π 4.1 π 4.2 π 4.3 π 4.4 π 4.5 π 4.6
π 5.1 π 5.2 π 5.3 π 5.4 π 5.5 π 5.6
π 6.1 π 6.2 π 6.3 π 6.4 π 6.5 π 6.6
(1)
The Markov Model nature of medical service projects is confirmed by the fact that in both medical
projects and Markov chains there are transitions between states of the system step by step, there are
transitional probabilities between individual states, the sum of transitional probabilities of a state is
equal to one, the sum of probabilities of all states place similarity of topological structure of
transitions.
Compare the properties of treatment service projects and the obtained model to prove that the
system can be described using Markov chains [13].
The properties of medical service projects that correspond to the Markov chains include:
- operational actions in projects: a) random process; b) for the community of consumers of
services there is a certain set of conditions; c) it is not possible to take into account the prehistory
of the transition of the community of service consumers to a certain state; d) therapeutic measures
carried out at time tk, transfer the system to a new state;
- therapeutic measures correspond to the steps of the process;
- the result of treatment projects forms the distribution of probabilities of the states of the
community of service consumers, and it is possible to indicate the possible transitions of the
system from each state to another in one step;
- the probability of transitions to other states depends on the properties of the system in which
random processes operate [15];
- since the states of the community of consumers of medical services are a complete group, the
sum of these probabilities is equal to one;
- transitions from any state of the system to other states constitute a complete group of events, one
of which must take place ;.
- the states of the system are displayed in a graph, indicating possible transitions from one state to
another in one step.
The analysis of the properties of the object and the model allow us to draw a conclusion about the
validity of the use of Markov chains for modeling projects for the provision of medical services [16].
Various approaches can be used to determine the values πij. An effective way to find the transition
probabilities is the method of directly measuring the time spent on certain operations of a certain state
based on the “photograph” of the operations. It is assumed that the transition probabilities are
calculated as the ratio of the time allotted for each transition to the total operation time. This method
can be used in the construction of Markov chains for production operations and processes, which, as a
rule, are normalized, and the time of their execution is regulated by the relevant regulatory documents
(instructions, rules, regulations, etc). Transient probabilities can be determined using statistical data,
for example, the conditions of patients receiving medical services are clearly recorded in medical
records. In fact, such a method is identical to the direct measurement of the number of patients in
certain groups (conditions).
Transition probabilities, determined on the basis of expert assessments of specialists of the INTO-
SANA medical institution, are given below:
0,75 0,15 0 0,05 0,05 0
0,2 0,599 0,12 0,06 0,02 0,001
0,03 0,2 0,519 0,2 0,05 0,001
ij .
0,07 0,15 0,3 0,278 0,2 0,002
0,09 0 0,2 0,4 0,308 0,002
0 0 0 0 0,01 0,99 (2)
A property of the developed model is the dependence of the random process of change of Si states
in time t[0, T]. The value of s is a possible state of a random process Si(t), if in the interval [0, T] there
is such a time t that the probability P{s-z0. Time t runs a discrete series of
values t0, t1, t2, …, tN : {tn, n=0, …, N}and the random variable Si(tn)=Si|n can take a discrete set of
values s1, s2, …, sk або {sk, k= 1, …, K}.
It is known that when the initial state of the system is determined and for the matrix of transition
probabilities it is possible to find the probability of each of the states p1(k), p2(k),…p6(k) after any k
step.
m (3)
pi (k ) [ p j (k 1) ji ] ; i 1, 2,, n
m6
j 1
The obtained probabilities of conditions as a result of the performed medical measures allow to
predict and evaluate the efficiency of medical institutions. The results of changes in the probabilities
of the state of the community of consumers of medical services in steps are shown in Fig. 3. These
results reflect the current level of medical services, which is now characterized (in the quasi-
stationary position in step k = 40) by the following distribution of probabilities of conditions: : p1 (40)
= 0,30; p2 (40) = 0,26; p3 (40) = 0,19; p4 (40) = 0,14; p5 (40) = 0,08; p6 (40) = 0,03.
With the help of the developed model it is possible to estimate how the condition of consumers of
medical services on various influences and projects, including in the conditions of the organization of
insurance medicine will change.
Figure 3: Changing the probabilities of the state of the health care system: p1 (k) - almost healthy;
p2 (k) - able-bodied; p3 (k) - temporarily incapacitated; p4 (k) - chronic disease; p5 (k) - critical state;
p6 (k) - death.
In practice, when assessing the effectiveness of medical services, there are six health conditions,
one of which is likely to be every patient. Therefore, the proposed Markov model (6S) includes the
following generally accepted states: S1 - almost healthy; S2 - able to work; S3 - temporarily
incapacitated; S4 - chronic disease; S5 - critical condition; S6 - output (lethal outcome). There is a
certain system of connections between these states - transitions (Fig. 4).
Figure 4: The oriented graph of states of 6S’s model
For example, if the patient is in state S2, then in the case of the disease it is possible to transition
not only to state S3, but also to other states such as S4 or S5. Such a division of systems into certain
states with given transitions between them is inherent in Markov chains.
A matrix containing all the transition probabilities πij of the Markov chain shown in Fig. 4, has the
form:
1.1 1.2 1.3 1.4 1.5 1.6
2.1 2.2 2.3 2.4 2.5 2.6
3.1 3.2 3.3 3.4 3.5 3.6
ij .
4.1 4.2 4.3 4.4 4.5 4.6 (4)
5.1 5.2 5.3 5.4 5.5 5.6
6.1 6.2 6.3 6.4 6.5 6.6
The analysis of the properties of the object and the model allow us to draw a conclusion about the
validity of the use of Markov chains for modeling projects for the provision of medical services
(Table 1).
Comparison of the properties of the original - the community of patients and the Markov model
allows us to conclude that the identity of the characteristic properties that are important for solving
problems of modeling the state of the patient community in the provision of medical services. Model
identification indicates the possibility of applying Markov chain theory to model medical projects.
4. Experimentation
Table 1
Identification of properties of Markov chains of medical projects for the provision of medical
services
Properties Markov chain Medical projects
1. Transitions from the real A random process has a Service delivery process:
state to the future feature: for each moment of a) random process;
time t0 the probability of any b) the community of
state of the system in the patients can be divided into
future (at t> t0) depends only certain conditions;
on its current state and does c) it is impossible to take
not depend on when and how into account the prehistory of
the system came to this state. transitions between states;
d) services provided at time
t0, transfer the system to a
new state.
2. Correspondence of steps In a Markov chain with The medical services provided
discrete time the transition correspond to the process
from some state to other steps that bring the system to
states is carried out by process a new state.
steps.
3. The presence of a transition Transition probabilities Medical services change the
probability depend only on the state from probabilities of patient
which and to which the communities, and the
transition is made probabilities of transition to
other conditions depend on
the quality of service delivery.
4. Condition for the sum of The sum of probabilities of all The conditions of the patient
probabilities of states of the states of the Markov chain at community are a complete
system each step is equal to one. group, so the sum of the
probabilities of these
conditions is one.
5. Condition for the sum of The sum of transient Transitions from any state of
transient probabilities from probabilities from some state the system to other states
any state of the Markov chain to other constitute a complete group of
states is equal to one. events.
6. Topological similarity The states of the Markov chain The states of the system are
are represented by a graph, represented by a graph, where
which indicates the possible the edges are possible
transitions between states in transitions from one state to
one step. another in one step.
The transition probabilities πij, determined on the basis of expert assessments of specialists are
given below:
0,75 0,15 0 0,05 0,05 0
0,2 0,599 0,12 0,06 0,02 0,001
0,03 0,2 0,519 0,2 0,05 0,001
ij . (5)
0,07 0,15 0,3 0,278 0,2 0,002
0,09 0 0,2 0,4 0,308 0,002
0 0 0 0 0,01 0,99
The obtained probabilities of conditions as a result of the performed medical measures allow to
predict and evaluate the efficiency of medical institutions. The matrix of transition probabilities πij
reflects the existing level of medical services, which is now characterized for k = 40 by the following
distribution of probabilities of states of consumers of medical services: p1(40) = 0,30; p2(40) = 0,26;
p3(40) = 0,19; p4(40) = 0,14; p5(40) = 0,08; p6(40) = 0,03.
The quality of medical care affects the conditional probabilities of transitions from S3 to states
S2 and S4 (Fig.4). Possible intervals of change π32 = 0 ... 0,6. This means that in the case of high quality
treatment for temporary disability up to 60% of patients can go into S2. In the current situation, this
value is 20%. At the same time, with high quality treatment, the value of π43 should decrease.
Therefore, for the study, we take π43 = 0.1 and determine the probability distribution of states when π32
changes (Fig. 5). The introduction of quality medical care will lead to a change in some transitional
probabilities, which will significantly change the general picture of the state of consumers of medical
services.
In fig. 5 presents the results on the change of probabilities of the states of the system of medical
services at different values of the transition probability π32, which characterizes the level of quality of
medical services. An increase in π32 from 0.15 to 0.6 leads to a significant decrease in the probability
of p3 (40), which characterizes the temporary incapacity of health care consumers. This increases the
probabilities of p1 (40) and p2 (40), which correspond to the share of practically healthy and able-
bodied consumers of medical services. It is these consumers of services who are solvent and can form
the financial basis for the provision of medical services.
Forecasting the effectiveness of projects being developed is rationally performed using
probabilistic models that reflect the specifics of random processes. The random nature of the demand
for medical services is obvious, which allows us to present the activities of medical institutions using
the Markov model.
а) π32=0,15
b) π32=0,60
c) π32=0,45
d) π32=0,60
Figure 5: Changing the probabilities of the state of the health care system under conditions of
improving the quality of treatment: p1 (k) practically healthy; p2 (k) able to work; p3 (k) temporarily
incapacitated; p4 (k) chronic disease; p5 (k) critical state; p6 (k) death.
Conclusion
The design of medical services is the most important condition for the successful implementation
of medical activities, which is usually considered the art of the doctor. But the management of
medical services also contains an organizational and technical component - planning, implementation
of treatment projects, control, analysis and correction of results. When managing health care projects,
one of the main tasks is to assess the effectiveness of projects.
The analysis of the considered approaches to definition of efficiency of medical projects with
reflection of specificity of casual processes allows to define use of modern mechanisms of modeling
of Markov's model of changes. The current WHO model allows to evaluate only the qualitative results
of a medical project to promote a specific medical service, but does not reflect quantitative results and
random processes. To solve this problem, the authors used modeling of Markov processes.
This approach allowed to build a basis for predicting the effectiveness of medical projects, in
particular in insurance activities.
5. References
[1] V. Gogunskii, O. Kolesnikov, G. Oborska, , ... S. Harelik, , D. Lukianov. Representation of
project systems using the Markov chain, in: Eastern-European Journal of Enterprise
Technologies, volume: 2(3-86), 2017, pp. 60-65. doi:10.15587/1729-4061.2017.97883
[2] V. Morozov, O. Mezentseva, M. Proskurin. Application of game theory for decisions making
on the development of it products, in: Advances in Intelligent Systems and Computing,
volume: 1, 2021. doi: 10.1007/978-3-030-54215-3_24
[3] S. Chernov, S. Titov, L. Chernova, ... L. Chernova, K. Kolesnikova. Algorithm for the
simplification of solution to discrete optimization problems, in: Eastern-European Journal of
Enterprise Technologies, volume 3(4-93), Kharkiv, 2018, pp. 34-43. doi:10.15587/1729-
4061.2018.133405
[4] D. Shanling, W. Zheng-Guang, S. Peng, S. Hongye. Quantized Control of Markov Jump
Nonlinear Systems Based on Fuzzy Hidden Markov Model, in: IEEE Transactions on
Cybernetics, volume: 49 , issue: 7, 2019, pp. 2420 – 2430. doi: 10.1109/TCYB.2018.2813279
[5] K. Senthuran, R. Sithamparanathan, S. Evans. Markov Decision Process-Based Opportunistic
Spectral Access, in: IEEE Wireless Communications Letters, volume: 5 , issue: 5, 2019, pp.
544-547. doi: 10.1109/LWC.2016.2600576
[6] N.M. Ghahjaverestan, S. Masoudi ; M. B. Shamsollahi ; A. Beuchée ; P. Pladys ; D. Ge ; A.
I. Hernández. Coupled Hidden Markov Model-Based Method for Apnea Bradycardia
Detection, in: IEEE Journal of Biomedical and Health Informatics, volume: 20, issue: 2,
2016, pp. 527-538. doi: 10.1109/JBHI.2015.2405075
[7] M. Roveri. Learning Discrete-Time Markov Chains Under Concept Drift, in: Transactions on
Neural Networks and Learning Systems, volume: 30, issue: 9, 2019, pp. 2570-2582.
doi: 10.1109/TNNLS.2018.2886956
[8] H. Liu, L. Yang, J. Chen ; M. Ye ; J. Ding ; L. Kuang . Multivariate Multi-
Order Markov Multi-Modal Prediction With Its Applications in Network Traffic
Management, in: IEEE Transactions on Network and Service Management, volume: 16,
issue: 3, 2019, pp. 828-841. doi: 10.1109/TNSM.2019.2934133
[9] Tzortzis, C. D. Charalambous, T. Charalambous, C. N. Hadjicostis, M. Johansson.
Approximation of Markov Processes by Lower Dimensional Processes via Total Variation
Metrics, in: IEEE Transactions on Automatic Control, volume: 62, issue: 3, 2017, pp. 1030-
1045. doi: 10.1109/TAC.2016.2578299
[10] C. Snider, J. A. Gopsill, S. L. Jones, L. Emanuel, B. J. Hicks. Engineering Project Health
Management: A Computational Approach for Project Management Support Through
Analytics of Digital Engineering Activity, in: IEEE Transactions on Engineering
Management, volume: 66, issue: 3, 2019, pp. 325-336. doi: 10.1109/TEM.2018.2846400
[11] K. Kanaev ; A. N. Ivanin. On Approaches to the Functioning of Transport Communication
Networks Modelling in the Context of Network and Computer Attacks Based
on Markov Processes Theory, in: 2020 Wave Electronics and its Application in Information
and Telecommunication Systems (WECONF), 2016, pp. 325-336.
doi: 10.1109/WECONF48837.2020.9131431
[12] J. Xie, W. Gao, C. Li. Heterogeneous network selection optimization algorithm based on
a Markov decision model in: IEEE Transactions on Engineering Management, 2018, volume
2, pp. 40-53. doi: 10.23919/JCC.2020.02.004
[13] R. Zhang, Y. Yu, M. El Chamie, B. Açıkmeşe and D. Ballard. Decision-making policies for
heterogeneous autonomous multi-agent systems with safety constraints, in: Proc. 25th Int.
Joint Conf. Artif. Intell., pp. 546-552, 2016.
[14] N. Demirer, M. El Chamie, B. Açıkmeşe. Safe Markov chains for ON/OFF density control
with observed transitions, in: IEEE Trans. Autom. Control, volume: 63, no.: 5, 2018, pp.
1442-1449. doi: 10.1109/TAC.2017.2755366
[15] B. Açıkmeşe, N. Demir and M. Harris. Convex necessary and sufficient conditions for density
safety constraints in Markov chain synthesis, in: IEEE Trans. Autom. Control, volume: 60,
no.: 10, 2015, pp. 2813-2818. doi: 10.1109/TAC.2015.2400712
[16] E. Altman and A. Shwartz. Markov decision problems and state-action frequencies, in: SIAM
J. Control Optim., volume: 29, no.: 4, 1991, pp. 786-809.
[17] X. Ding, S. L. Smith, C. Belta and D. Rus. Optimal control of Markov decision processes
with linear temporal logic constraints, in: IEEE Trans. Autom. Control, volume: 59, no.: 5,
2014, pp. 1244-1257. doi: 10.1109/TAC.2014.2298143
[18] E. Feinberg and A. Shwartz, Handbook of Markov Decision Processes: Methods and
Applications, New York, NY, USA:Springer-Verlag, 2002.
[19] E. A. Feinberg and A. Shwartz, Constrained Markov decision models with weighted
discounted rewards, in: Math. Oper. Res., volume: 20, no.: 2, 1995, pp. 302-320.
[20] M. L. Puterman, Markov Decision Processes: Discrete Stochastic Dynamic Programming.
New York, NY, USA:Wiley, 1994.