Current oil and gas market trends, characterized by great competitiveness and increasingly complex contradictory constraints, have pushed researchers to design an adaptive control system that is not only able to react effectively, but is also able to adapt to rapidly evolving demands at a fixed cost. The system does this by using the available resources as efficiently as possible to optimize this adaptation. [4] presented an analysis of the needs of production systems, highlighting the advantages of adopting a self-organized heterarchical control system. The term, heterarchy, is used to describe a relationship between entities on the same hierarchical level [6]. Initially proposed in the field of medical biology, it was then adapted for several other domains [9; 10; 7]. In the multi-agent domain, the term, heterarchy, is relatively close to the concept of "distribution", as used in "distributed systems". However, from our point of view, the fact that the decisional capacities are distributed does not mean that the multi-agent system is organized heterarchically, even though this is often the case [15;17]. Nonetheless, the heterarchic organization of distributed systems is the assumption that we make in this paper. From our point of view, this assumption is justified by the system dynamics and the volatility of the information, which make a purely or partially hierarchical approach inappropriate for creating an effective reactive system [4].
In this paper, we focus on the dynamic control of complex manufacturing systems, such as those found in the petroleum industry. In this industry, all functions (e.g., planning, scheduling and maintenance) and resources (e.g., turbines, storage systems) are automated.
This unit is designed to produce oil from the base oil treated in the units HB3 and HB4 and imported additives, the base oil is received in Tank TK2501 to TK2506. Each docking Tank stock defined grade of oil (SPO, SAE10-30, BS) (Production of 132,000 t / year for an amount of 10% additives) if the type of oil stored in a tank must be changed, the tank must first be rinsed for hours which is often avoided. This unit produces two major oil: engine oils 81% of the production (gasoline, diesel, transmission oils) and industrial oils (hydraulic (TISK), turbines (torba), spiral (Fodda), compressor (Torrada) and various oils). To do this, two methods are used: continuous mixing (mixing line) and mixing in discontinuous (batch) (see Figure 1). In this article we focus on the mixing line. To produce finished oil, a recipe must be applied:
X1% Hb1 + X2% Hb2+ X3% Additif1
Where : Xi is the rate and HBi is the base oil.
The mixing line its base oil from the docking Tanks, which produce this decade plan (see figure 2):
In this paper, we aim to develop an adaptive control system for Unit3100 which will produce dynamically efficient scheduling solution using resources in optimal way.
We consider each resource and Oil in tank as a decisional entity, and we model them as agents.
We conducted a state-of-the-art review of the dynamic scheduling problem in the literature. This section highlights the studies that reflected our point of view.
In manufacturing control, scheduling is the most important function. In this paper, we focus on dynamic scheduling.
[5] Have classified dynamic scheduling into three categories: predictive, proactive, and reactive. The first, predictive, assumes a deterministic environment. Predictive solutions call for a priori off-line resource allocation. However, when the environment is uncertain, some data (e.g., the actual durations) only becomes available when the solution is being executed. This kind of situation requires either a proactive or reactive solution. Proactive solutions are certainly able to take environmental uncertainties into account. They allocate the operations to resources and define the order of the operations, though, because the durations are uncertain, without precise starting times. However, such solutions can only be applied when the durations of the operations are stochastic and the states of the resources are known perfectly (e.g. stochastic job-shop scheduling) [3]. The third type of dynamic scheduling, reactive, is also able to deal with environmental uncertainties, but is better suited for evolving processes.
Reactive solutions call for on-line scheduling of resources. In fact, the resource allocation process evolves, making more information available and thus allowing decisions to be made in real-time [16; 11; 5; 1]. Naturally, a reactive solution is not a simple objective function, but instead a resource allocation policy (i.e., a state-action mapping) which controls the process. In this paper, we focus exclusively on reactive solutions.
Over the last few decades, scheduling researchers were inspired by artificial intelligence whose methods were based exclusively on operational research algorithms of exponential complexity. Taking into account performance effectiveness and efficiency, which means optimizing several criteria, will increase problem complexity even more. Artificial intelligence has allowed such complex problems to be solved, yielding satisfactory, if not always optimal, solutions.
[9] used genetic algorithms (GA) to adapt the decision strategies of autonomous controllers. Their control agents use pre-assigned decision rules for a limited amount of time only, and obey a rule re-placement policy that propagates the most successful rules to the subsequent populations of concurrently operating agents. However, GA do not provide satisfactory solutions for reactive scheduling. Therefore, a reactive technique must be integrated into GA to allow the system to be controlled in real time.
Reinforcement learning (RL) might be an appropriate way to obtain quasi-real-time solutions that can be improved over time. Reinforcement learning is learning by trial and error dedicated to agents learning. In this paradigm, agents can perceive their individual states and perform actions for which numerical rewards are given. The goal of the agents is thus to maximize the total reward they receive over time. [8] used reinforcement learning to optimize resource use in a very expensive electric motor production system. Such systems are characterized by a variety of products that are produced on re-quest, which requires a great deal of flexibility and adaptability. The assembly units must be autonomous and modular, which makes performance control and development difficult. [8] considered these units as insect colonies able to organize themselves to carry out a task. Self-organization can reduce the number of resources used, allowing production risk problems to be solved more easily.
The most used reinforcement learning algorithm is Q-learning. [18] extended this algorithm by using a reward function based on EMLT (Estimated Mean LaTeness) scheduling criteria, which are effective though not efficient. [2] pro-posed an intelligent agent-based scheduling system. They employed the Q-III algorithm to dynamically select dispatching rules. Their state determination criteria were the queue's mean slack time and the machine's buffer size. These authors take advantage of domain knowledge and experience in the learning process.
But in this paper, we are exploring a more developed algorithm "SARSA algorithm" in a heterarchical organisation of agents. In conclusion, we are trying to experiment reinforcement learning by using SARSA algorithm to conceive an adaptative and reactive manufacturing control system for petroleum process based on heterarchical multi-agent architecture. In the next section, we will present our system architecture and motivating our choices.
A multi-agent system is a distributed system with localized decision-making and interaction among agents. An agent is an autonomous entity with its own value system and the means to communicate with other such entities. For a general survey of the application of multi-agent systems in manufacturing, see the review by [1]. In order to develop multi-agent system with a reactive decision capability in an uncertain environment, they may be modelled as Markov Decision Process (MDP) [12]. And to improve the system performances and learn optimal policy in Markov environment, If the transition function T (modelling the system's evolution from state to state) is unknown while an objective can be identified a learn-by-trial process such as RL [12;13] can be designed.
We consider that a petroleum refinery exists in a dynamic, uncertain and unpredictable environment, since it is subject to internal stress (e.g., production risks) and external constraints (e.g., forced markets, unexpected orders). According to [12], the decisions made in such environments involve Markov decision processes (MDP). Clearly, in such a Markovian context, it is necessary to consider the transition function T, modelling the system's evolution from state to state as an unknown. According to [12] and [13], a learn-by-trial process, such as reinforcement learning, should be used determine the optimal policy. This modelling approach is widespread. Figure 1 shows the main functions embedded in each agent.
An MDP is a tuple < S,A,T,R >, where S is a set of problem states, A is a set of actions, T(s, a, s') [0, 1] is a function defining the probability that taking action a in state s results in a transition to state s', and R(s, a, s') R defines the reward received after such a transition.
If all the parameters of the MDP are known, an optimal policy can be found by dynamic programming. If T and R are initially unknown (which is commonly the case when considering industrial case studies), Reinforcement learning (RL) methods can learn an optimal policy by direct interaction with the environment. RL is learning to act by trial and error. Agents perceive their individual states and perform actions for which numerical rewards are given. The goal of the agents is thus to maximize the total reward received over time. This technique is often used in robotics, in order to teach a robot the behavior to achieve its goals and to overcome obstacles.
The SARSA algorithm is used to learn the function Q π (s, a), defined as the expected total discounted return when starting in state s, executing action a and thereafter using the policy π to choose actions:
The discount factor γ ∈ [0,1] determines the relative importance of short term and long term rewards. For each s and a we store a floating point number Q(s,a) for the current estimate of Q π (s,a).
As experience tuples < s,a,r,s',a' > are generated through interaction with the environment, the table of Q-values is updated using the following rule:
The learning rate α ∈ [0,1] determines how much the existing estimate of Q π (s,a)
contributes to the new estimate.
If the agent's policy tends towards greedy choices as time passes, the Q(s,a) values will eventually converge to the optimal value function Q*(s,a). To achieve this, we use a Boltzman probability which determines the probability of choosing a random action.
Figure 2 shows the steps of the SARSA algorithm Fig. 2. The SARSA algorithm
In our case, this algorithm will make the Resource Agent learn its action policy π, which in turn makes it able to choose the best action for each state (accept task/request, or not). This algorithm works with the following data:
State parameters are the current time t ∈0…T; the inventory of pmps p 1 … p n and their states Sp 1 … Sp n (e.g., maximum capacity, feeding, receiving); the list of Storage Tanks T 1 … T m , and their states ST 1 …ST m (e.g., Capacity). Action concerns the reception or not of the product, stop or start pumping.... Reward function assigns no reward to most of the states and positive rewards to a specific goal state. For more precision and to obtain a proper convergence, the reward function is a state combination engendered by an action. One idea was to take into account the volum in tanks and (C i ) and feeding and uploading stream (Fd i ) (Ud i ) in the reward function:
As shown in Figure 3, the MCSR (Manufacturing Control System using Reinforcement learning) architecture consists of "resource agents" for the pumps, "parts agents" for the tanks containing oil and an "observer agent" to control the process.
Based on agents ha knowledg reference containin the state v The idea is roughly the following: a part agent has a task request that it proposes to resource agents, and then the resource agents give their propositions. The part agent chooses the best proposition and establishes the contract. A detailed illustration of the agent interaction is provided in figure 4.
Our model was simulated in the Borland Jbuilder environment because of its potential for facilitating communication and thread programming and because of its compatibility with the chosen MADKIT platform architecture for SMA development (visit http:// www.madkit.org/downloads). One of the advantages of the reinforcement learning algorithms is that they allow evaluation during learning. To permit this evaluation, we selected the following criteria.
A petroleum refinery is subjected to many operational constraints. Operational constraints include the requirement that only one tank at a time can receive oil, but several can simultaneously feed mixing line, and another that states a tank cannot receive and send oil at the same time. Problem inputs include the base oil arrival schedule, which describes the volumes and qualities of the base oils and additives that will be received in the refinery during the desired time horizon; the finished oil demands, and the current levels and qualities of the base oil in the storage tanks. The major constraints considered can be formalized as follows (see parameter definitions given in 4.2):
C1: Tank storage level can never be less than a given threshold ( ) min
The base oil is stored in specific storage tanks (TK2501-TK2506 (see figure 5)). The total time horizon spans 160 hours, during which completely defined oil parcels have to be received from the pipeline. Six oil tanks are available; all of them have the same capacity, but different amounts of oil at the beginning of the time horizon (figure. 6.) Aims are to receive all base oil using available pumps feeding Tank with sufficient capacity, and to produce exactly the requested quantity with the available quantity of bases oil in the range of the decade. For this reason, we consider as an evaluation criterion the Cmax (Maximum duration time to produce the requested products).
The experiment was conducted as follows: we launch the system with data explained above. The graph (Figure 7) shows the results for the first phase of the learning algorithm. As this graph shows, before 5000 iterations, the Cmax variation is rather high. It varied in the interval [100h, 1500h], which is a modest result. This can be justified by the fact that the results are from the exploration phase, in which actions are executed randomly according to the Boltzmann probability [1]. The second phase is the exploitation phase, in which the choice of actions is based on Q values (just before and after 5000 iterations), and the results are better. This phase produced solutions with a very interesting Cmax of 45 h. Thus, we can state that our system converges towards optimal solutions by minimizing the total time of production even with maintenance tasks.
Despite being relatively under control, thanks to the preventive maintenance plans, perturbations are always possible in a refinery. To test our system faced with such random events, we caused system perturbations in order to observe the system's behavior. We caused the same perturbation (a breakdown of P3102) in the exploration phase at the 2000th iteration and again in the exploitation phase at the 15000th iteration. When such perturbations occur in the current system, some production tasks have to be cancelled to allow the maintenance tasks to be performed. The human expert then has to manually find a solution to replace the cancelled production tasks. However, in our experiment, the disturbance in the exploitation phase was quickly compensated for without any Cmax variation over 49h, and the system was brought back to the level of its best performances. These results show that our system is able to learn how to establish a continuously improving optimal control policy to schedule maintenance tasks within a production plan without reducing the production rate.
In this paper, we have presented a multi-agent model for the dynamic scheduling of in petroleum process. In this model, agents simultaneously insure effective scheduling and continuous improvement of the solution quality by means of reinforcement learning, using the SARSA algorithm. We have also provided an overview of the research done in the field of manufacturing control, focusing on dynamic and reactive scheduling. The results of our experiments with this model show that our approach can generate on-line scheduling solutions and improve their quality by minimizing Cmax. Nevertheless, we want to widen the time horizon of our experimentation, taking into consideration more complex production units. Last, we are going to work on a holonic version of our model for future comparison with the multi-agent model.