-In a not so far future, private houses will be provided with devices that can produce renewable energy, and this will give the owners the chance of selling the unused energy to neighbors. The fact that this selling will be based on peer to peer negotiation (i.e., between single producers and single consumers), will make this market deregulated. This situation could lead to advantages for both producers, who will have an extra income for energy they would not use, and consumers, who can buy cheaper energy than the big companies' one. However, this scenario is very complex and dynamic, and without an appropriate management can lead to odd situations. This paper presents the agent-based modeling of an application to manage the negotiation among different parties producing and consuming energy. We will show that the feature of autonomy of agents well suit the requirements of the proposed scenario. Moreover, we will exploit game theory to define a strategy that try to optimize energy production and supply costs by means of negotiation and learning. By means of simulation of the different parties we will show the effectiveness of the proposed approach; the results show that applying our approach enables to reduce the price of the energy and leads to an equilibrium between expected and real prices.
The growing diffusion of solar panels and wind turbines not only in public places but also in domestic environment is driving the evolution of the energy market. In fact, while today’s energy market is characterized by a centralized approach, i.e. few energy provider companies regulated by the governments, in the future we expect that also small and home energy producers can play an important role in this market. On the one hand, this kind of producers rely on renewable and clean sources; on the other hand, unused energy will not wasted but sold to consumers.
This evolution towards deregulation should take into
consideration also Transmission and Distribution System Operators
(TSOs and DSOs), which might extend their services to
domestic users who rely on renewable energy devices in order
to let them participate as consumers and at the same time
producers, forming a new form of actor (the prosumer). This
scenario introduces new paradigms of price profiles [
These changes can be thought in the context of the Smart Grid, giving the possibility to ordinary consumers to retrieve their needed energy from the neighbouring prosumers that can supply both those domestic environments. This will create a new kind of decentralized distribution net involving several kinds of sellers with added dynamism and a finer time granularity for contracts’ stipulation.
The contribution of this paper is to propose a specific agent oriented architecture in which software agents represent different types of users: from the ordinary energy consumer represented by a buyer agent, to the newly-introduced prosumer agent, but also considering agents acting on behalf of traditional big energy companies. Agents, thanks to their autonomy, well suit the requirements of the depicted scenario. Our proposal exploits also game theory, and in particular the class of minority game was considered and adapted to this project.
In the design of the proposed architecture we took into
account several aspects, such as system heterogeneity, reliability,
scalability and security. We have simulated the architecture in
order to show the effectiveness of our approach, but we have
also implemented it in order to show the practical applicability
of the approach; in particular, we have exploited JADE1 as
agent platform to implement the architecture and to perform
our tests. Such tests have to deal with all the most important
aspects of the energy problem: balancing electricity demand,
forecasting supplies and also negotiation with specific market
adaptation strategies. Readers interested in the he metering
aspect, can find in [
The paper is organized as follows. After reviewing the related literature (Section II), the models and agents used are presented (Section III). In Section IV the market adaptation algorithm will be presented, showing simulations and results in Section V. Final remarks and future work possibilities end the paper in Section VI.
Interesting applications featuring agents in the energy
market can be found in [
Definitions and notations for the game theoretic concepts
commonly used later in this work can be found in [
In this section we give a complete overview of the agents set required in the energy trading scenario proposed describing the kind of agents involved in the energy market with a in depth explanation of the most important steps of their behaviors. In addition to that, a clear distinction of how an agent is supposed to act in an hypothetical real environment and what actually performs the simulation software, needs to be highlighted in order to obtain a better understanding of the problem. The platform used for this application has different kinds of agents according to their roles. Main and auxiliary agents are present, as explained as follows:
Buyers are energy consumers and they usually outnumber
the sellers; they do not produce energy so they are searching
for obtaining their electricity demand supplied by stipulating
contracts related to a specific time interval. Each market day
is divided into several time intervals and for each one every
buyer has to decide in advance who is going to be its energy
supplier for the next time interval. In the developed software,
a balancer agent controls the amount of energy exchanged in
the negotiation process (the details are explained later in this
section). Buyers can predict how much energy they need for
the following time interval. This can be obtained by reading
previous electric measurement and by applying an energy
consumption forecasting algorithm. It is important to perform this
forecast before any negotiation, so that the buyer can choose
the most suitable seller according to the energy availability
of the suppliers. A really effective forecasting algorithm that
fits our short-term paradigm is thoroughly described in [
A buyer can stipulate a contract with a prosumer after winning an auction round, based on sealed bids; a prosumer can sell its energy to more than one buyer, while, for simplicity’s sake, we assume that one buyer buys energy from one prosumer only. For a prosumer once the investment in a small-scale energy production plant based on renewables is realized, any positive amount derived by selling energy contributes to the investment return. Therefore in order to be attractive, prosumers’ starting prices can be considered substantially lower than Gencos’ initial contract prices. Prosumers communicate to buyers an initial starting price that is influenced by contracts with DSOs/TSOs and a random cost due to the devices used to produce electricity (e.g., maintenance costs). The energy produced by a prosumer has to be sold and cannot be stored or buffered. Every prosumer is in direct competition with other sellers: they have to propose an appealing starting price and make an intelligent use of refusing bids in order to rise the price and, at the same time, avoid pushing buyers in contacting other sellers. Gencos are big energy generating companies. They have a theoretically infinite amount of energy supply, but sold at a fixed price, so there is no auction negotiation and every contract can be stipulated much faster compared to the prosumers’ auction system.
However their prices are higher than prosumers’ starting price and they depend on TSO/DSO contracts, raw material prices and (most important in our scenario) threshold exceeding costs. This aspect is thoroughly explained in the following paragraph and represent a modeling choice to prevent overloading production lines as well as avoiding concentrating a huge number of consumers for a single big producer. A Genco receives a request from a buyer; then it just calculates the price according to the above-explained variables and communicates the final price back to the buyer.
Gencos’ threshold system. A key point is how much energy a generating company can produce without having to buy a quantity on the market (e.g., a foreign and more expensive market) or switching to more polluting production lines. Thus we assume that every Genco has a supply threshold, and once reached, the Genco has to buy energy Cu = abroad (the energy production of that seller is under stress). So the energy cost can be calculated as follows: (Costenergy if below supply threshold
where Cu is a single energy unit cost, EC > 1 is an external cost constant and A > 0 is the number of energy units above the threshold.
In addition, surpassing the threshold might also be harmful
for the environment since more polluting plants might be
started (e.g., oil based). Asking the Genco for contracts when
this threshold is already surpassed leads to more expensive
contract prices. Those prices rise as we get further from the
specified threshold. This particular pricing strategy already
introduced in [
Other auxiliary agents, not directly involved in the negotiation process are represented by the Balancer and the Time agents. While the latter’s only duty is to provide a time reference for synchronizing processes, the Balancer agent is responsible for the demand/supply balancing aspects: it acts in the very first step of the negotiation round by retrieving the single demand of every consumer and the production forecasts of the prosumers.
Having a clear understanding of the balancing needs of the
grid is essential. In fact, recent studies [
The data in Fig. 1 refers to GridView2 production cost model, with hourly load, solar and wind projections for 2030 based on 2006 information to maintain data correlation. On a separate note, it is important to point out that, in Fig. 1, solar plants have production peaks during central hours of the examined days.
In our model we are clearly dealing with the (b) situation
when it comes to balancing issues. Several mathematical
models are presented, but most of them are different way to set
to zero the algebraical sum between demand on one side and
2http://www.abb.com/industries/
supply to the other side [
Out simplified mathematical approach is explained in the following.
by that specific Genco
provided by that specific Prosumer
DC tk the demand of the Kth buyer
The ability of producing an amount of energy is influenced mostly by the market of raw materials for the Genco production line, while the prosumers have to deal with local weather.
Producing more than the quantity that they are supposed to supply is risky for the sellers since we assume the absence of buffering or storing of surplus energy. Moreover, we have to take into account all the previous considerations regarding traditional suppliers versus PVs and CSPs.
Demand D is calculated by a specific algorithm of demand forecasting, but no matter which kind of statistics we are going to use in order to solve that, we have to specify that the demand refers to a pre-determined interval of time. That is because we are trying to deal with the short term market paradigm in order to avoid overloading electric lines as a result of bad long term forecasting or unoptimized distribution.
Obviously D is just the sum of all the demands (at a certain time) needed for all the consumers in the area. It has to satisfy the balance relationship in equation 1:
Equation 1 does not take into account unavoidable leaks and calculating errors. On the other side, if the supply and demand forecasting are efficient and precise enough, we can rely on an easy implementation model for simulations. Equation 1 is quite straightforward in its meaning: the sum between the two production sources (Gencos and prosumers) should be equal to the total consumer demand. Also, from previous sections, we know that dealing with a fixed demand will cause the other two elements to change accordingly and it is more likely to see in the future an increment on the prosumers’ supplies that will be balanced by a decrease of gencos’ production.
Other agents used for simulation purposes are represented by an Agent Creator who is able to dispatch the other agents in the respective areas [17] and an exception handler agent used to increase performances, which has been implemented during scalability and reliability tests [18].
In order to provide a clearer picture on how the contract negotiation and the adaptation to the energy market has been modelled for our test simulations, this section presents an overview of the behavior that agents are following during a single negotiating round. Some auxiliary agents have been left out, due to their simple tasks that does not require further explanations, while the Balancer and Prosumer’s behavioural steps are shown in Fig. 2. Also the Genco has been left out due to the simplicity of its behavior compared to buyers/prosumers auction system and due to the fact that its threshold pricing model has already been thoroughly explained.
The steps in Fig. 2 provide a complete picture on what happens during a single negotiating round. Some behaviors are common, such as the discovery of agents according to the role they have: this is obtained using a feature of the chosen agent platform. In fact, JADE has a distributed Directory Facilitator (DF) in which any agent can register itself to be then found by other agents distributed elsewhere, therefore the DF acts as a yellow pages service. The registration itself is not shown in Fig. 2, since just the steps in the JADE main behavioural method (i.e: action()) is shown. Registration in the DF is done just once in the initializing method, while the search and discovery is done in every negotiating interval. This latter choice obviously introduces more computational load, however it is completely justified for having a dynamic architecture in which the number of total peers is constantly changing. An introduction of a new seller, for instance, will be known to the other agents starting from the following time interval.
Initial steps indicate the retrieving of web services for both consumers and sellers. The goal of the former is to obtain the local temperature to know in advance if an air conditioning system will be active: thinking about a function for describing the bound between temperature and the consequent energy consumption, we can roughly describe a V shaped function in which to the lowest amount of energy used corresponds to average temperature from 19 to 21 degrees, while we have consumptions peak as far as we move from this point (meaning that is either too hot or too cold). An agent can retrieve temperature values using appropriate web services and a prosumer does the same for obtaining weather information for forecasting its production (e.g: wind direction and strength in case it has a micro generation through wind turbine).
Still regarding the buyer’s initial steps, a buyer can retrieve
informations about previous consumptions and also on going
tariffs by interacting with a Smart Meter (a new generation
electric energy consumption reader). This has been previously
and successfully tested with this presented implementation [
Concerning the buyer’s market strategy and adaptation to the dynamics of the short term electricity contracts, in Fig.2, we can see how the agents’ decisions are taken in different steps: as soon as they have received the notification from the balancer to start the negotiation, they have to first decide to contact a prosumer or a Genco. This is done by using a minority game derived algorithm (see Section IV), taking into account the limited prosumers supplies compared to the traditional Gencos. In case of the choice of contacting a prosumer, also the amount of stakes and maximum number of sent bids follow the adaptation algorithm: the goal is to avoid wasting time in sending multiple bids while Gencos are exceeding their production threshold. The market adaptation deals with the last step: every consumer has to evaluate if his budget expectations have been respected, changing how to rise their bids accordingly to the previous negotiation outcomes. This latter step is obtained by an added fuzzy logic block.
The buyer agents in our architecture are supposed to chose between contacting a Genco or a prosumer when they first receive a notification for beginning the negotiation. This two path approach has to lead to the cheapest energy contract possible for the buyer that acts on behalf of an human user and therefore has to simulate his/her rationality. Having a restricted set of actions as initial choices and a final outcome to be evaluated suggests us to seek in the game theoretic literature for a similar scenario that we can apply for solving our problem. In particular in the class of minority games we can think about a scenario in which two actions are initially possible and the outcome of this game depends on the actions of the other players, provided that each participant does not know in advance how his competitors will act. In these games, the players who have chosen the action taken by the minority of the total participants are rewarded with higher payoffs. Furthermore, since the energy market model proposed repeats itself in several negotiating round we should also take into account the game theoretic notion of repeated game. We are now presenting an already solved basic game scenario and on a second step we will show how to extend it to provide our buyer agents with an adapting strategy for the presented market model. A. El Farol Bar game
“El Farol Bar” is an existing bar situated in New Mexico
(USA). Every Thursday night it delivers discounted drink
prices, becoming really appetizing for the local potential
costumers, making obvious why every person living near the
bar, wants to go there on that particular night. The bar has
been used to model the El Farol Bar minority game [
Switching back to our problem, the two possible initial choices of action are still present in our energy related scenario: if every agent contacts a Genco, it will result in overloading the production lines of these big energy producers, causing them to provision in more expensive markets with high prices for the end-user and environmental issues too. Likewise, if every agent contacts (or tries to do so) the same restricted set of prosumers, only a few number of participant gets a nice deal, due to the fact that a prosumer can deliver a little amount of energy, especially compared to a Genco.
In our problem, we can adapt the different degrees of payoff of the bar scenario with the difference between what a single agent expected to spend and what it actually spends at the end of the negotiation interval (budget evaluation).
A simple way to find an equilibrium for the El Farol Bar
game has been proposed originally in [
According to the demonstration in [
Action Contact Pros.
Contact Genco
P. has supplies See TAB3 (+2 Ip) See TAB4 (+1 Ip)
P. has no supplies See TAB2 (0 Ip) See TAB4 (+1 Ip) symmetrical mixed strategy solution:
M H Where p is the probability to go at the bar and M , L and H the payoffs as shown in Table I.
Following studies (i.e., [
Using equation 2, we can see that for each participant we have a given probability that can be used to decide whether it is advisable to attend the discounted price night. Repeating the game we can see that every agent sooner or later will attend the bar and that most of the times, the pub will not be so crowded.
When trying to apply the solution shown in the equation 2 to our energy problem, we map some variables as follows:
T for the ratio between the amounts of energy produced by Prosumers over the total production, N is the total number of buyer agents and M , H and L are intervals defined according to the expected/actual money spent. A difference between the bar game and our energy market is that in the bar game if a number m of people are attending the bar with m > T then m players are losing. In our problem just T m people are actually going to retrieve a low payoff.
This initial model still lacks of influential variables like time constraints and limited prosumers’ supplies, implying the necessity of adding further stages to our game. We now present an approach in which several tables represent different payoff matrices for all the stages forming the game. This new methodology that mixes the minority game approach with a stochastic game (every payoff table refers to a specific participant’s state) is used in order to model the complexity of the energy problem.
The main idea behind the adaptation of the game we propose is presented more formally in Fig. 3. It is an infinite game split into finite rounds. The decision each agent takes at every state is compactly represented in the following payoff tables.
Tables II and III are called initial state tables while Tables IV and V are defined as final state tables. The difference is that only Tables IV and V show an ending of the negotiation, represented by the letters H , M or L as the payoff entity inside those cells.
Every buyer starts by taking a decision in the first table (referring to an element of the state space M). The balancer agent is the entity that knows how much energy can be Let I be a set of agents representing the consumers; P be the set of prosumers, while G represents the Gencos; Players 2 I move through different tables shaping the finite state space M = fm0, m1, m2, m3g. m0: initial state in which the agent i 2 I decides who is going to first contact. It can be a Genco or a specific Prosumer P0 2 P. m1: second state in which i decides who will be contacted next, provided that D(i) > S(P0) with D being the consumer demand and S is the seller’s supply capacity. m2: here i decides if it is convenient to bid a previously contacted prosumer Px 2 P or abort the negotiation, given D(i) S(P0). m3: i decides to accept or not the offer of a Genco Gx 2 G. Therefore each player (agent) i 2 I can perform an action inside the set(s): Ai(m0) = Ai(m1) = fContact Genco, Contact Prosumerg; Ai(m2) = fPlace bid, Abort Negotiationg; Ai(m3) = fAccept offer, Refuse offerg.
The probability P to move from the current state mx to next state (my) after performing a specific action a 2 A, written P (mx,a,my) is described in Tables II, III, IV and V with their assigned payoff chains.
Fig. 3. Game formalization. produced by all the prosumers and by using this information it can calculate the number of buyers that could be served by prosumers; this number can be related to the threshold T in the El Farol game. According to that threshold we can calculate the probability to contact prosumers instead of a Genco in this stage of the negotiation (quite similar to how it was possible to solve the “El Farol Bar” dilemma using the unique mixed strategy solution). However, at this moment we do not have a clear vision of future payoffs, but we can assign to those initial tables a certain amount of fictional points that we call “Intermediate points” (Ips). Those Ips represent the chain of payoffs for the stochastic game approach: assuming that every action taken by a participant agent is time consuming, decreasing Ips simulates time flow as well as a risk increase that the participating agent should be aware of. Risk awareness in auction bidding systems has already been studied [19]; although the concept of risk is elaborated in a different kind of market model, a risk-aware agent better simulates how a human user would act. On the other side, higher Ips increase the chance to have a satisfactory game result (H or M final payoff). In this way the buyer is redirected to other tables until it reaches a final cell: doing so the number of Ips can increase in case it is a lucky choice (contacting a prosumer that for sure has enough supplies) or decrease in the opposite scenario. In the initial state tables the buyer is redirected to other tables according to a previously calculated value that is related to the amount of energy all prosumers can produce. In the final state tables the algorithm is different: in order to simulate the importance of the time variable, lower Ip values mean that the buyer has been travelling around different tables for such a long time and chances to find a suitable seller or even a Genco that has not overtaken its threshold will be scarce. That is because in the ending tables negative values are present. When the Ip value is very small (<< 0) then the agent is forced to get a contract with a Genco in order to avoid wasting other time (and consequently other money).
At the end of each round, each buyer agent evaluates its outcome. Above we said that the difference between the expected money spent and the actual money spent can point out who are the winners and who are the losers, however ending a negotiation in a H(L) payoff cell of a matrix not always ensure a win(lose) situation: that is because the market dynamics (being bounded by swinging prices of raw energy production materials) leads to constant changes in energy minimum prices. Therefore obtaining an higher contract price compared to the pre-determined budget can happen even if the agent ended its cycle with an H payoff: in this case it just means that the agent was expecting an unrealistically low contract price. The same considerations have to be done in the opposite scenario of having an L payoff for a cheap contract. This implies the addition of a fuzzy logic block able to adjust the expected budget and the amount for the single bids (in case of a prosumer auction). The further we are from the centre of the fuzzy logic function, the stronger will be the reaction of the agent (either to increase or decrease stakes and/or expected budget), following simple and linear trends.
In such complex and dynamic scenario, a simulation is needed to prove if the designed strategy could be used by agents to negotiate in the market, thus obtaining cheaper contract prices. In particular, we use 5 consumers, 3 prosumers and 2 Gencos within a 10 round negotiation runs to test the JADE agents implementation. This simpler scenario allows us to evaluate the game based algorithm with different price scales using agents. The restricted number of agents, does not compromise the purpose of the test: this is because the kind of market modelled is more heavily influenced by the ratio between total demand and prosumers’ supplies rather than the number of agents per se.
Several parameters can be adjusted influencing the agent decision, namely: (1) number of Ips used as threshold in order to redirect the participant from one final table to the other; (2) difference between starting prices for the two kinds of sellers; (3) threshold switching values in the fuzzy logic block; (4) best way to assign values to H, M and L final payoffs; (5) price dynamics from one round to the other; (6) Gencos’ price penalties for exceeding thresholds; (7) probability for a prosumer to become more expensive than a Genco; and (8) accuracy about energy supply and demand forecasting that might not be 100% correct.
The best way to give a precise value to these parameters is to study an analytical formulation in which we can combine all the other known values (e.g., number of participants and amount of demands and supplies) in order to retrieve the unknown constants. However, due to the complexity and dynamics of the proposed model, we decided to use a numerical approach by trying several value combinations of every input variables of the algorithm.
At the end of each round, the buyer agent calculates the average expecting budget and the average money spent, assigning to each round number those other two values (e.g., round #, Paid Price, Expected Price).
In order to have a clearer idea of the efficiency and precision of the strategy, we show the difference between applying the presented algorithm or use a baseline set of actions. In the latter scenario, every buyer will contact a prosumer straightaway, since their starting prices are lower, becoming more appetizing to a rational agent. In addition, after signing a contract, the participant does not adjust any strategy parameter.
We obtain the results shown in Fig. 4, under the following conditions: (1) intersection between average starting prices of the sellers should not exceed 33%; (2) slow and not exaggerated price swings between each round; (3) significant price penalties for exceeding Gencos’ threshold; (4) the higher the error percentage between the forecast demand values and the actual requested values (negative error), the better becomes the improvement between using the presented algorithm compared to the baseline scenario; positive errors may worsen participant performances; and (5) very fast reaction to follow the expected price. The conditions (1) and (3) force the gap between the prices to be wide enough to justify the minority game approach, while (2) and (5) deal with the difficulty of the algorithm in finding equilibria in exaggerate dynamic scenarios. While (4) is straightforward.
The results of the simulation, as depicted in Fig. 4, show that expected prices follow the previous peak of paid prices. It is important to highlight that we are also trying to simulate the impact of swinging prices due to raw material prices fluctuations and/or payback costs for solar panels or wind turbines for prosumers. Even if those swings are not exaggerated due to high granularity for stipulating contracts, they are indeed an additional challenge to further prove the reaching of certain equilibrium scenarios. Proving the effectiveness of the described game is a challenging open question that we tried to answer with this simulation test.
Fig. 4. Prices varying during 10 rounds with and without the presented algorithm (JADE output). Prices have to be intended as price per energy unit. In the simulation, the expected price starts from 0 in the first round, reaching a convergence during the 8th round. Starting from that point, it becomes visible how expected and obtained prices of agents that follows the minority and stochastic approach (represented by the two continuous lines in Fig. 4), will constantly chase each other. Economically speaking, it means that in earlier rounds a buyer agent adopting the algorithm with the described strategies is likely to pay equally or slightly more than an agent following other strategies. However, if we consider a sufficiently large number of rounds, the saving compared to agents following the baseline behaviour (Fig. 4, the dotted line) is obtained more frequently, with significant lowest peaks during the most expensive period for buying energy.
The test was executed having a constant numbers of agents, although sellers’ supply capacity was subject to randomized swings from one round to the other. Therefore, changing sellers’ number does not drastically affect the presented results, provided that this number does not exaggeratedly and unrealistically change in a short period of time. In a more complex scenario in which sellers adopt strategies according to the economic background, the presence of different market competitors will determine an additional factor that needs to be further investigated in order to provide a more realistic model.
Computationally wise, the complexity of the presented algorithm is variable but does not appear to represent a problem. While the balancer agent has the duty to solve equation 2, buyer agents just have to solve an iterated amount of conditional instruction and comparing variables (e.g: if current Ip value is greater than the threshold value then execute action A, otherwise jump to action B). The fuzzy logic block is just composed of a mixed set of linear functions and it is executed just once at the end of the negotiating round. On a separate note about the architecture itself, further tests and improvements on performances (also in a distributed net) are considered in [18].
In this paper we have presented an agent-based architecture for deregulated energy market. We discussed aspects like balancing, pricing, negotiation and adaptation, which were taken into consideration for modelling, implementing and simulating the architecture. Fig. 4 shows that the adoption of an adaptive strategy produce better results when certain conditions are satisfied. Moreover, the expected prices, starting from very low values, tend to reach an equilibrated amount that represents the cheapest alternative in almost all the examined negotiation rounds. Agents that always try to win prosumers’ auctions may have some chance to win during the initial rounds, but still the algorithm provided tries to establish a Nash equilibrium nonetheless; once the prices are balanced, chances to obtain the best bargain are going to be sporadic for those agents. [17] N. Capodieci, “P2P energy exchange agent platform featuring a game theory related learning negotiation algorithm,” Master’s thesis, University of Modena and Reggio Emilia, 2011, available at http://www.cs.rug.nl/ aiellom/tesi/capodieci.pdf. [18] M. Koster, “Reliable Multi-agent System for a large scale distributed energy trading network,” Master’s thesis, University of Groningen, 2011, available at http://www.cs.rug.nl/ aiellom/tesi/koster.pdf. [19] V. Robu and H. L. Poutre´, “Designing bidding strategies in sequential auctions for risk averse agents,” In Proc. of AMEC07, 2007.