Petroleum industry production systems are highly automatized. In this industry, all functions (e.g., planning, scheduling and maintenance) are automated and in order to remain competitive researchers attempt to design an adaptive control system which optimizes the process, but also able to adapt to rapidly evolving demands at a fixed cost. In this paper, we present a multi-agent approach for the dynamic task scheduling in petroleum industry production system. Agents simultaneously insure effective production scheduling and the continuous improvement of the solution quality by means of reinforcement learning, using the SARSA algorithm. Reinforcement learning allows the agents to adapt, learning the best behaviors for their various roles without reducing the performance or reactivity. To demonstrate the innovation of our approach, we include a computer simulation of our model and the results of experimentation applying our model to an Algerian petroleum refinery.
Current oil and gas market trends, characterized by great competitiveness and
increasingly complex contradictory constraints, have pushed researchers to design an
adaptive control system that is not only able to react effectively, but is also able to
adapt to rapidly evolving demands at a fixed cost. The system does this by using the
available resources as efficiently as possible to optimize this adaptation. [
In this paper, we focus on the dynamic control of complex manufacturing systems, such as those found in the petroleum industry. In this industry, all functions (e.g., planning, scheduling and maintenance) and resources (e.g., turbines, storage systems) are automated. 2
This unit is designed to produce oil from the base oil treated in the units HB3 and HB4 and imported additives, the base oil is received in Tank TK2501 to TK2506. Each docking Tank stock defined grade of oil (SPO, SAE10-30, BS) (Production of 132,000 t / year for an amount of 10% additives) if the type of oil stored in a tank must be changed, the tank must first be rinsed for hours which is often avoided. This unit produces two major oil: engine oils 81% of the production (gasoline, diesel, transmission oils) and industrial oils (hydraulic (TISK), turbines (torba), spiral (Fodda), compressor (Torrada) and various oils). To do this, two methods are used: continuous mixing (mixing line) and mixing in discontinuous (batch) (see Figure 1). In this article we focus on the mixing line. To produce finished oil, a recipe must be applied:
X1% Hb1 + X2% Hb2+ X3% Additif1
In this paper, we aim to develop an adaptive control system for Unit3100 which will produce dynamically efficient scheduling solution using resources in optimal way. We consider each resource and Oil in tank as a decisional entity, and we model them as agents. We conducted a state-of-the-art review of the dynamic scheduling problem in the literature. This section highlights the studies that reflected our point of view. In manufacturing control, scheduling is the most important function. In this paper, we focus on dynamic scheduling.
[
Reactive solutions call for on-line scheduling of resources. In fact, the resource allocation process evolves, making more information available and thus allowing decisions to be made in real-time [16; 11; 5; 1]. Naturally, a reactive solution is not a simple objective function, but instead a resource allocation policy (i.e., a state-action mapping) which controls the process. In this paper, we focus exclusively on reactive solutions.
Over the last few decades, scheduling researchers were inspired by artificial intelligence whose methods were based exclusively on operational research algorithms of exponential complexity. Taking into account performance effectiveness and efficiency, which means optimizing several criteria, will increase problem complexity even more. Artificial intelligence has allowed such complex problems to be solved, yielding satisfactory, if not always optimal, solutions.
[
Reinforcement learning (RL) might be an appropriate way to obtain quasi-real-time solutions that can be improved over time. Reinforcement learning is learning by trial and error dedicated to agents learning. In this paradigm, agents can perceive their individual states and perform actions for which numerical rewards are given. The goal of the agents is thus to maximize the total reward they receive over time.
[
The most used reinforcement learning algorithm is Q-learning. [
But in this paper, we are exploring a more developed algorithm “SARSA algorithm” in a heterarchical organisation of agents. In conclusion, we are trying to experiment reinforcement learning by using SARSA algorithm to conceive an adaptative and reactive manufacturing control system for petroleum process based on heterarchical multi-agent architecture. In the next section, we will present our system architecture and motivating our choices.
A multi-agent system is a distributed system with localized decision-making and
interaction among agents. An agent is an autonomous entity with its own value
system and the means to communicate with other such entities. For a general survey
of the application of multi-agent systems in manufacturing, see the review by [
We consider that a petroleum refinery exists in a dynamic, uncertain and
unpredictable environment, since it is subject to internal stress (e.g., production risks)
and external constraints (e.g., forced markets, unexpected orders). According to[
An MDP is a tuple < S,A,T,R >, where S is a set of problem states, A is a set of
actions, T(s, a, s’)Æ [
Performances
If all the parameters of the MDP are known, an optimal policy can be found by dynamic programming. If T and R are initially unknown (which is commonly the case when considering industrial case studies), Reinforcement learning (RL) methods can learn an optimal policy by direct interaction with the environment. RL is learning to act by trial and error. Agents perceive their individual states and perform actions for which numerical rewards are given. The goal of the agents is thus to maximize the total reward received over time. This technique is often used in robotics, in order to teach a robot the behavior to achieve its goals and to overcome obstacles. The SARSA algorithm is used to learn the function Qπ(s, a), defined as the expected total discounted return when starting in state s, executing action a and thereafter using the policy π to choose actions:
Qπ (s, a) = ∑ T (s, a, s′)[R(s, a, s′) + γ Qπ (s′,π (s′))] (1)
s′
The discount factor γ ∈ [
As experience tuples < s,a,r,s’,a’ > are generated through interaction with the environment, the table of Q-values is updated using the following rule:
Q(s, a) = (α − 1)Q(s, a) + α (r + γ Q(s′, a′)) (2)
The learning rate α ∈ [
If the agent's policy tends towards greedy choices as time passes, the Q(s,a) values will eventually converge to the optimal value function Q*(s,a). To achieve this, we use a Boltzman probability which determines the probability of choosing a random action. In our case, this algorithm will make the Resource Agent learn its action policy π, which in turn makes it able to choose the best action for each state (accept task/request, or not). This algorithm works with the following data: State parameters are the current time t ∈ 0…T; the inventory of pmps p1… pn and their states Sp1… Spn (e.g., maximum capacity, feeding, receiving); the list of Storage Tanks T1… Tm, and their states ST1…STm (e.g., Capacity). Action concerns the reception or not of the product, stop or start pumping.... Reward function assigns no reward to most of the states and positive rewards to a specific goal state. For more precision and to obtain a proper convergence, the reward function is a state combination engendered by an action. One idea was to take into account the volum in tanks and (Ci) and feeding and uploading stream (Fdi) (Udi) in the reward function: ⎧ 1 if Ci (t) = C maxi ⎪ RPart−Ag = ⎨ 0 if Ci (t) ≥ C mini ⎪ ⎩−1 if Ci (t) < C mini
⎧ 6 ⎪1 if ∑ ⎪ i=1 RRe source− Ag = ⎨ 6 ⎪−1 if ∑ ⎪⎩ i=1
Fdi = 1500 m3 / h
Fdi = 0
As shown in Figure 3, the MCSR (Manufacturing Control System using Reinforcement learning) architecture consists of “resource agents” for the pumps, “parts agents” for the tanks containing oil and an "observer agent" to control the process.
Based on Alaadin modelling (Ferber and Gutknecht, 1998), the resource and parts agents have certain properties, roles and groups. Initially, each agent must have knowledge about its properties (e.g., tank number, capacity, characteristics… or pump reference, flow stream…), its role (i.e., storage or pumping) and its group (e.g., tanks containing the same product). The observer agent has a global view of the system, and the state variables that it observes are the indicators of performances.
Fig. 3. MCSR architecture (Manufacturing Control System using Reinforcement learning)
The Observer Agent receives the decade production demand. It sends the relevant set of tasks to each agent. Each part agent (i.e., tanks containing oil) and resource agent (i.e., pump) perceives its state which is a combination of its individual state (e.g., stopped, busy) and the set of tasks that they must execute.
To deal with agent interaction, we used the well-known Contract Net protocol [
The idea is roughly the following: a part agent has a task request that it proposes to resource agents, and then the resource agents give their propositions. The part agent chooses the best proposition and establishes the contract. A detailed illustration of the agent interaction is provided in figure 4.
Our model was simulated in the Borland Jbuilder environment because of its potential for facilitating communication and thread programming and because of its compatibility with the chosen MADKIT platform architecture for SMA development (visit http:// www.madkit.org/downloads). One of the advantages of the reinforcement learning algorithms is that they allow evaluation during learning. To permit this evaluation, we selected the following criteria.
A petroleum refinery is subjected to many operational constraints. Operational constraints include the requirement that only one tank at a time can receive oil, but several can simultaneously feed mixing line, and another that states a tank cannot receive and send oil at the same time. Problem inputs include the base oil arrival schedule, which describes the volumes and qualities of the base oils and additives that will be received in the refinery during the desired time horizon; the finished oil demands, and the current levels and qualities of the base oil in the storage tanks. The major constraints considered can be formalized as follows (see parameter definitions given in 4.2): C1: Tank storage level can never be less than a given threshold Ci (t) ≥ C mini C2: Tank storage level can never be greater than a given threshold . Ci (t) ≤ C maxi C3: mixing line must always contain oil ∑n Fdi (t) > 0 i=1 ⎧Udi (t) > 0, Fdi (t) = 0 C4: Tank cannot feed and receive at the same time ⎨ ⎩Udi (t) = 0, Fdi (t) ≥ 0 The base oil is stored in specific storage tanks (TK2501-TK2506 (see figure 5)). The total time horizon spans 160 hours, during which completely defined oil parcels have to be received from the pipeline. Six oil tanks are available; all of them have the same capacity, but different amounts of oil at the beginning of the time horizon (figure. 6.) Aims are to receive all base oil using available pumps feeding Tank with sufficient capacity, and to produce exactly the requested quantity with the available quantity of bases oil in the range of the decade. For this reason, we consider as an evaluation criterion the Cmax (Maximum duration time to produce the requested products). 5.2
The experiment was conducted as follows: we launch the system with data explained
above. The graph (Figure 7) shows the results for the first phase of the learning
algorithm. As this graph shows, before 5000 iterations, the Cmax variation is rather
high. It varied in the interval [100h, 1500h], which is a modest result. This can be
justified by the fact that the results are from the exploration phase, in which actions
are executed randomly according to the Boltzmann probability [
We caused the same perturbation (a breakdown of P3102) in the exploration phase at the 2000th iteration and again in the exploitation phase at the 15000th iteration. When such perturbations occur in the current system, some production tasks have to be cancelled to allow the maintenance tasks to be performed. The human expert then has to manually find a solution to replace the cancelled production tasks. However, in our experiment, the disturbance in the exploitation phase was quickly compensated for without any Cmax variation over 49h, and the system was brought back to the level of its best performances. These results show that our system is able to learn how to establish a continuously improving optimal control policy to schedule maintenance tasks within a production plan without reducing the production rate.
In this paper, we have presented a multi-agent model for the dynamic scheduling of in petroleum process. In this model, agents simultaneously insure effective scheduling and continuous improvement of the solution quality by means of reinforcement learning, using the SARSA algorithm. We have also provided an overview of the research done in the field of manufacturing control, focusing on dynamic and reactive scheduling. The results of our experiments with this model show that our approach can generate on-line scheduling solutions and improve their quality by minimizing Cmax. Nevertheless, we want to widen the time horizon of our experimentation, taking into consideration more complex production units. Last, we are going to work on a holonic version of our model for future comparison with the multi-agent model.