Thermal storage is an essential concept within many energy systems. Such storage is generally used in order to smooth out the time lag between the acquisition and the use of energy, for example by using heat water tanks within heating systems. In this work we use a multi-agent system in order to maintain and operate distributed thermal storage among a large group of buildings in a district heating system. There are several financial and environmental benefits of using such a system, such as avoiding peak load production, optimizing combined heat and power strategies and achieving general energy efficiency within the network. Normally a district heating system is purely demand driven, resulting in poor operational characteristics on a system wide scale. However, by using the thermal inertia of buildings it is possible to manage and coordinate the heat load among a large group of buildings in order to implement supply driven operational strategies. This results in increased possibilities to optimize the production mix from financial and environmental aspects. We present a multi-agent system that combines the thermal storage capacities of buildings with production optimization strategies. The system consist of producer agents responsible for valuing the heat load management, consumer agents managing the quality of service in individual buildings while consenting to participate in heat load management, and a market agent acting as a mediating layer between the producer and consumer agents. The market agent uses an auction-like process in order to coordinate the heat load management among the consumer agents, while the producer agents use load forecasting in order to evaluate the need for heat load management at any given point in time. A consumer agent uses continues feedback regarding indoor climate in order to uphold quality of service while participating in heat load management. Real-time data from a district heating system in Sweden is used in order to evaluate the agent system in relation to operational peak load management. The results show clear financial and environmental gains for the producer as well as participating consumers.
This paper describes a multi-agent system used in order to improve the operational
management of a district heating system in relation to financial and environmental
aspects. District heating and cooling is an integrated part of the energy infrastructure
in many countries. This is especially true in Northern and Eastern Europe, although
district heating and cooling is growing rapidly in popularity around the world [
Normally a district heating system is purely demand driven, in the sense that the production units can only react to the current heat demand within the network. The bulk of the heat demand consist of demand for building heating, although substantial heat load peaks are also generated by social patterns in relation to tap water usage. The heating demand in a building is correlated to the outdoor temperature, while tap water usage is more correlated to time of day. This combination of outdoor temperature dependencies and social behaviour over the day causes fluctuations in the heat demand. Such fluctuations or heat load peaks are very undesirable for a number of reasons. Most district heating systems have a production set-up using some sort of base load boilers which are complemented with a range of peak load boilers. The base load boilers are often fuelled by cheap and environmentally friendly fuels such as biomass or industrial waste heat, while the peak load boilers normally use fossil fuels such as oil or gas which are not only expensive but also generate undesirable emissions.
In this paper we describe a multi-agent system which uses the thermal inertia of
buildings in order to manage the heat load demand in relation to the operational status
at the production units. By managing the demand in such a way it is possible to make
the district heating system more supply driven, i.e. it is possible to optimize the heat
load usage in relation to the production status and not only the other way around. This
is similar to using large storage tanks, although here the thermal inertia of the
connected buildings is used as a distributed storage instead. By using such a set-up it is
possible to shed heat load peaks in order to avoid using peak load boilers, or to move
the heat load peaks in time in order to coordinate the heat load peaks with high spot
prices for electricity when using CHP. The use of large storage tanks is well
understood and is a mature technique within district heating systems, and similar strategies
are possible to utilize with the proposed distributed thermal storage [
The multi-agent system consists of three different agent entities; the producer agent, the consumer agent and the market agent. This paper focuses on the market agent component which acts as a mediating layer between the consumer and producer agents. The market agent uses as an auction-like process in order to coordinate heat load demand among the consumer agents in relation to the operational status of the producer agents.
The paper starts off by describing related and previous work concerning multiagent based heat load management within district heating systems. In section three the physical process of managing the heat load usage in relation to thermal inertia of a building structure is discussed. Section four presents the functionality of the coordination process within the multi-agent system. Section five describes the experimental set-up used in evaluating the system, and the results are presented and discussed in sections six and seven. Finally the paper is concluded in section eight and nine with conclusions and future work. 2
The framework for the multi-agent systems is built around three different agent
types; the consumer agents, the producer agents and the market agent used to
coordinate the interaction between the two former agent groups. This basic set-up was first
introduced in a previous paper [
The consumer agent tries to balance the ability to partake in load control while
simultaneously upholding required levels of quality of service (QoS). The concept of
quality filters was proposed in [
The basic process of using heat storage facilities within buildings in order to
improve the operational functionality of a district heating system has been previously
studied in [
An early attempt on demand side management was presented during the 1980s.
And although the current hardware and communication capabilities limited the
operational functionality of the proposed system, the future potential of the system was
apparent [
The multi-agent system uses three different main types of agents in order to distribute and coordinate the heat load among the participating buildings; producer agents, consumer agents and market agents. A producer agent will work to optimize its operational production in regards to fuel prices in the general case and power market spot prices in the case of combined heat and power generation. Such operational management is done by distributed heat storage among buildings connected to the district heating network. Whenever the producer agent deems it necessary to perform heat load management it will request load control among the consumer agents. The producer agent will have to pay the participating consumer agents so it is important for the producer agent to set the price constraints correctly. The coordination of this load control between the consumer and production agents is done by the market agent.
The agents are controlled by an implicit norm system in which agents are assumed not to cheat or lie. Specifically, in order to avoid collusive behaviour the agents are not permitted to form coalitions. A consumer agent is assumed to be truthful regarding its internal energy balance status, i.e. it is assumed that a building owner doesn’t want the residents to complain about the indoor temperature. Furthermore, in multiproducer agent settings it is assumed that the producer agents will not impose on load management instigated by other producer agents, although they have the information to do this based on the on-line auction status.
An energy company and all its production units are represented by a producer agent. Most district heating systems are run by one single energy company, but there could also be several energy companies competing within a network. In the case of a single energy company the producer agent will only need to consider its own operational constraints, while in a multi-energy company setting it will also need to perform real-time evaluation in regards to the status of its competitors.
Each consumer sub-station participating in the system will be represented by a consumer agent. Normally each building has its own sub-station, but several buildings can also be connected to a single sub-station. A consumer sub-station consists of heating control systems, heat exchangers, valves and pumps, and this is the point of control for the consumer agent. Normally such a sub-station will be connected to an outdoor temperature sensor which acts as the input signal for the control system, i.e. the colder it gets, the more the heat demand will increase. In order to control the heat load demand of the building a Linux-based computer I/O platform is used. This platform is connected between the outdoor temperature sensor and the existing control system. By manipulating the outdoor temperature signal the consumer agent is thus able to influence the control behaviour of the sub-station. Figure 1 shows an example of how this works in practice.
A B C D
E
F
The data in Figure 1 is collected from an online consumer agent operating on the I/O platform. The data shown covers twenty-four hours. The forward temperature into the heating system is shown by the line A, while the return temperature from the heating system is shown at line B. Together with the mass-flow (not shown in the picture), the amount of energy used can be calculated. Line C shows the average indoor temperature of the building in question. Line D shows the difference between the forward and return temperatures in the heating system, i.e. line A minus line B. Line E shows the actual outdoor temperature, while line F shows the manipulated outdoor temperature. In the example it is shown that E and F are identical until the consumer agent performs a short load control, when E and F diverge in the middle of the figure. When this happens the forward temperature into the heating system (A) is lowered, which leads to a decrease in heat load instantaneously. Conversely the opposite can be done if the consumer agent wants to buffer the thermal inertia of the building by inserting extra heat.
Unlike the producer and consumer agents there is no physical object corresponding to the market agent. However, there is a good reason to separate this functionality since it will make for a more transparent coordination process. Since actual money is involved in the process it is important that the market process is deemed trustworthy by all participating entities. 3.1
! ! !!! !!!
In a previous paper we presented an optimization algorithm for synchronizing heat load usage in relation to power market spot prices. It was shown that such optimization resulted in substantial financial benefits when using combined heat and power generation on a day-ahead and intraday spot price market. In this paper we extend this algorithm to cover the general case, i.e. adapting it for distributed heat storage in general. The generalized optimization model is shown in equations 1 to 7. max
!(!!,!) ∗ (!"#$%! − !"#!!,! − !"!,!) !" ! !!!!,! = 0 !ℎ!" ! !!,! = 0
!"#$%&' !" !(!) = 1 !" !!,! ≠ 0 0 !" !!,! = 0
!!""#$ ≥
!(!!,!) !!"#$% ≤
!(!!,!) ! !!! ! (eq. 6) !(!!,!) ≤
! !!,!!! + !!"#$%&' !(!!,!) ≥ ! !!,!!! − !!"#$%&' (eq. 7) (eq. 8)
Equation 2 maximises the total earnings. The value of sold heat can only change from hour to hour, while the cost for producing the heat can change both in time and in relation to the level of heat load demand. The costs are separated in fuel costs and operational costs. Such operational costs include costs for starting and shutting down boiler and costs related to managing the boilers. An m by n matrix represents the heat load demand during a time period, where m is the theoretical maximum amount of heat load in the system. The value of n is normally 24, i.e. the head load demand is evaluated one day ahead and is discretizised into hours. Equations 3-8 define the operational boundary conditions for the optimization model.
An optimization model for production is dependent on heat load forecasts ranging from hours to a few days ahead. Such forecasts are calculated based on a combination of previous operational data and weather forecasts. The main parameter from the weather forecast to consider is the outdoor temperature, although general weather conditions such as precipitation and wind affect the situation to some extent. Using the heat load forecasts a producer agent can arrange the wanted heat load demand hour by hour in relation to fuel prices and other related costs. By doing this it is possible to minimize the use of expensive and environmentally unsound peak load fuels. In order to make the model useful in practice the heat load demand represented by a matrix which is discretizised into blocks. Each such block is called a heat block, and it is these heat blocks that the producer agent will sell to the consumer agents through the market process.
A heat block represents a certain monetary value for the producer agent, since if the consumer agents successfully implement the load control specified by the heat block this will translate into an improved operational status at the physical production units. However, some of these added earnings will have to be used to pay the consumer agents participating in managing the heat load demand within each heat block. Based on the optimization model the producer agent will be able to evaluate the financial value for heat block, and will thus be able to set a maximum price it is willing to pay for the implementation of the associated load control. The producer agent is also able to set the size for the heat blocks based on the boundary conditions in the optimization model, i.e. the size of the matrix is based on the operational conditions within the production units. 3.2
The consumer agents get paid by the producer agents in order to perform load control. The payments are accumulated during the payment period, usually a month, and then subtracted from the heating bill. Obviously a consumer agent would want to maximize this revenue, by performing as much load control as possible. At the same time the consumer agent is responsible for maintaining an acceptable level of quality of service, i.e. a building owner wants to save money on the heating, but only to the point where the residents of the building are not complaining about it. Therefore the consumer agent assets are modelled based on the indoor temperature, and thus the agent’s ability to accumulate heat load control through the auction process is inversely proportional to any deviation from the wanted indoor temperature. The consumer agents also has an upper and lower limit of maximum allowed deviation from the wanted indoor temperature, outside which the agent is not allowed to perform any load control at all. Furthermore the consumer agent also has time constraints in regards to how long the temperature is allowed to deviate from the wanted temperature, i.e. the agent cannot just set the temperature at the upper or lower limit and let it stay there indefinitely. All these operational constraints are set by the building owner depending on characteristics of the building in question.
In order to fulfil the task of maintaining quality of service the consumer agent must have some way of evaluating the indoor temperature. The I/O platform used in the installed system is equipped with wireless communication for such sensors which make it easy to collect indoor sensory data. In addition to this the consumer agent continuously calculates an internal energy balance model of the building based on sensory data from the consumer sub-station in relation to the energy signature and time constant of the building in question. The energy signature is a characteristic value for each individual building indicating the amount of heat load needed to uphold one degree of difference between the outdoor temperature and the indoor temperature. The time constant is another value specific for each building indicating how long time it takes for the indoor temperature to reach equilibrium with the outdoor temperature if no external heating is supplied. Using these values is convenient since they can normally be estimated within acceptable levels without any costly energy analysis of each building. It is also convenient in relation to the auction process since a change in the indoor temperature in relation to the specific energy signature of a building translates to a specific amount of heat load, while the time constant translates into an estimate of how long a specific heat load change can be upheld without jeopardizing the quality constraints. This makes it possible for the consumer agent to relate the indoor temperature status directly to an amount of available currency, expressed in the possible heat load change, to be used in the auction process. 3.3
When a producer agent estimates the need for load control it will issue one or more heat blocks to the market agent. Each such heat block is the size of the heat load times one hour, i.e. the length of time for a heat block is static while the size in heat load may vary. The market agent will then divide each heat block into heat slots which it auctions off among the consumer agents. This basic set-up is visualized in Fig 2.
The reason to divide the heat block into smaller slots is that normally no individual building is large enough to handle a whole block by itself. The market agent performs the sizing of these slots based on knowledge regarding the outcome of previous auctions, i.e. the market agent will have a rough idea about how large chunks the consumer agents can handle at a time. If the market agent does not have any such prior knowledge it will start by trying to auction off the whole block as one slot, then dividing it in half until consumer agents start to respond to the auction calls.
Each slot is sold through a reverse Dutch auction process. A Dutch auction starts at a high price which is then decreased in increments by the auctioneer until a buyer accepts the price. The purpose of using such auctions is mainly that they come to a conclusion fast, thus minimizing communication overhead which is imperative in an industrial setting such as this. A reverse Dutch auction is like a normal Dutch auction except that it starts from zero and works itself up until either a bid is accepted by some consumer agent or the maximum price set by the producer agent is reached. Once a bid is accepted the market agent will issue a contract to the winning consumer agent. The contract specifies the size of the heat slot and also the time when to start implementing the load control. A consumer agent can participate in several successive auctions, as long as it is able to maintain the quality of service.
A producer agent can issue a heat block many hours, or indeed days, before it is to be implemented. However, if the block is issued to early the consumer agents might be reluctant to commit themselves since they cannot predict their indoor temperature state too far into the future. On the other hand, if the heat block is issued to late a large portion of the consumer agents might already be under contract, potentially preventing them from participating further.
If the market agent isn't able to allocate the whole heat block, then the producer agent can choose to either accept the partially filled block or to reject it. In this case the consumer agents involved are bound by contract until the producer agent either accepts or rejects the block. On the other hand, if the market agent does succeed in allocating the entire heat block, the producer agent is obliged to accept it.
The market agent will keep track of the outcome of all auction activity. This data is publicly available to all entities participating in the agent system, which serves two purposes. First of all it ensures a transparency in the process, which is important since both consumer and producer agents want to be able to ensure the functionality of the debiting system. Secondly if such data is freely available all agents will share the same knowledge, thus decreasing the risk for unfairness in the market process. 4
The experiments in this project were performed using data from a district heating
system at Gothenburg Landvetter Airport in Sweden. This system has around thirty
buildings connected, and the size of these buildings range from large arrival halls and
air plane hangars to smaller buildings of various kinds. The data studied spans the
month of January 2011. This district heating system is installed with the previously
mentioned I/O platform, although the overall agent system was not yet operational at
the time of this study. Hence the agent coordination evaluation was done using the
DHEMOS simulation environment, although operational data from the district heating
system was used. DHEMOS is a simulation framework for district heating systems,
combining simulation models for production, consumption and distribution [
The scenario studied involves the management of peak load. The production units use a base load burner using biomass fuel and an oil based peak load burner. The oil burner is necessary to use at heat load demand levels above 4.5 MW. The goal is to minimize the use of the oil burner since it is both costly and environmentally unfriendly. The prices involved are based on averages during the month of January 2011. The fuel costs are valued at €32.29/MWh for biomass and €75,84/MWh for oil, while the income is €54,17/MWh. Biomass emit zero CO2 while oil results in emissions at 271 kg/MWh. 5
Fig 3 shows a comparison between DHEMOS and the measured operational data at the production site. The DHEMOS data has distributed heat storage (DHS) inactivated in order to compare to the operational data. During the first half of the month the operational data displays somewhat of a volatile behaviour in comparison with the simulated data. These violent fluctuations are caused my measurement errors. However, the general correlating trend between the measured data and the simulation is apparent. If the simulation data is used as a measurement of actual heat load demand it can been seen that short periods of heat load shortage has most likely resulted during the first and last peaks, at around 50 and 630 hours. The fact that no-one has complained during these times might be considered an indication in regards to the potential of thermal storage within buildings.
Fig 4 shows a simulation run with DHS activated in order to avoid peak load usage above the limit of 4.5 MW. The producer agents wants the heat load demand to be as close as possible to this limit, since they want to sell as much as possible of the energy produced by biomass.
It is obvious that the agent system isn’t able to fully avoid all peak load usage. During the latest peak load, at around 600 hours, the consumer agents are not capable of responding to the heat blocks being offered by the market agent. However during the major part of the month the system is able to successfully manage the heat load demand. At around 400 hours it can be seen how the consumer agents buffer heat by increasing their heat load demand in order to prepare for the coming peak at around 450 hours.
Fig 5 shows an example of the indoor temperature variations in a building during the simulation period. The consumer agent used the energy balance model in order to calculate this value.
The lowest temperature value accepted by the consumer agent in this case is nineteen degrees, while the wanted temperature is twenty-one degrees. The consumer agent approaches the lower limit on several occasions, most notably during the latest peak at around 650 hours. When this happens the consumer agent will start to lose auctions, thus rendering it unable to perform further load management. It should be noted the calculated indoor value shown above should be viewed as a control variable for the consumer agent, and not necessarily a representation of the actual indoor temperature. This is due to the fact that the energy balance model doesn’t take into account heat contributing factors such as solar radiation, excess heat from electronic equipment or social activities.
As the producer agent manages the heat load demand the total amount of energy sold will be reduced, but since the majority of this energy is produced by expensive oil the net profit will increase. Table 1 shows a comparison between active and nonactive distributed heat storage.
DHS off DHS on
3548,24 3245,62
192212 175819 Net 63905 69157
0% 30%
63905 65480
CO2 85540 11660 Table 1 show that the profit will increase by €3675 during the period and that the CO2 emissions will decrease by more than 86%. The agent cost represents the amount of money the producer agent uses to pay the consumer agents. In our simulations this value was between 10-30% although this is dependent on the ratio of competing consumer agents in relation to the amount of heat blocks available on the market, i.e. a direct relation between supply and demand. The cost of the installed system is about €20000 which in relation to a profit of about €3500 per month will lead to the system being amortized within one heating season. 6
The presented system uses an implicit norm system, which assumes that the agents are benevolent and acting towards globally known goals using universally accepted strategies. However, in a fully operational system these agents would be acting on a financial market including energy companies and building owners who want to maximize their own financial gains. In the current system norms are present in such a way that agents are expected to behave in a certain fashion, e.g. not to cheat or lie about their status. In a practical setting it is possible to maintain this system since the implementation of all agent parameters where set within this project. However, in a reallife system the participating building owners or energy companies would be able to set any parameters they like, and could even re-program the agents as long as they adhere to the communication protocol.
The strategy of the consumer agent is to bid as a combined function of its valuation of the load management and its estimation concerning other agents’ valuation. This obviously opens up for coalition forming among the consumer agents, although the current implementation doesn’t allow for this. Consumer agents might want to form coalitions in order to achieve high prices by agreeing to withhold auction bidding and then splitting the proceeds. If a group of consumer agents were successful in rigging the market like this they could achieve more than the 10-30% profit share they got during the presented experiments.
In regards to the producer agents there is no incentive for them to lower the maximum price below its true valuation, since this would only harm the producer agent itself by depriving it of financially profitable load management. 7
In this paper we have presented a multi-agent system for distributed heat storage. It was shown that such a set-up can be used to reduce, and in many instances remove, the need for financially and environmentally unsound peak load fuel usage. Since the multi-agent system is operationally adaptable it is able to adjust the heat load demand in relation to actual operational constraints among the production units. This makes it possible to increase the net profit for the energy company even though the total amount of energy being sold is less.
It was shown that the consumer agents will cooperate in load management as long as they are able to simultaneously uphold their desired quality of service.
The net profit in district heating system in question was increased by about 2.5% which translates to a return of investment in less than one heating season considering the installation costs involved. At the same time the CO2 emissions were reduced by more than 86% due to the shift in fuel composition. 8
In the future a framework for explicit management of norms will have to be added. This is imperative since the physical entities represented by the agents need to maintain trust in the system, even when those entities are allowed to set their own agent parameters. This system should be based on a set of explicit norms or ground-rules, which are then managed through a layer of individual trust among agents coupled with a globally visible reputation system. The market agent will act as the main manager for this system, and will use punishment by exclusion in order to maintain order. Exclusion translates into financial loss, which will act as deterrence for the agents to deviate from the norms. 9
The authors would like Swedavia for letting us use operational data from the district heating system at Gothenburg Landvetter Airport. Also, we would like to thank NODA Intelligent System AB for the use of the agent I/O-platform. 10