-This paper describes a multi-agent based simulation (MABS) framework to construct an artificial electric power market populated with learning agents. The artificial market, named TEMMAS (The Electricity Market Multi-Agent Simulator), explores the integration of two design constructs: i) the specification of the environmental physical market properties, and ii) the specification of the decision-making (deliberative) and reactive agents. TEMMAS is materialized in an experimental setup involving distinct power generator companies which operate in the market and search for the trading strategies that best exploit their generating units' resources. The experimental results show a coherent market behavior that emerges from the overall simulated environment.
The start-up of nation-wide electric markets, along with
its recent expansion to intercountry markets, aims at
providing competitive electricity service to consumers. The new
market-based power industry calls for human decision-making
in order to settle the energy assets’ trading strategies. The
interactions and influences among the market participants are
usually described by game theoretic approaches which are
based on the determination of equilibrium points to which
compare the actual market performance [
As an alternative to the equilibrium approaches, the
multi-agent based simulation (MABS) comes forth as being
particulary well fitted to analyze dynamic and adaptive systems
with complex interactions among constituents [
In this paper we describe a MABS modeling framework that provides constructs for the (human) designer to specify a dynamic environment, its resources, observable properties and its inhabitant decision-making agents. We used the framework to capture the behavior of the electricity market and to build a simulator, named TEMMAS (The Electricity Market Multi-Agent Simulator), which incorporates the operation of several generator company (GenCo) operators, each with distinct power generating units (GenUnit), and a market operator (Pool) which computes the hourly market price (driven by the electricity demand).
TEMMAS agents exhibit bounded rationality, i.e., they make decisions based on local information (partial knowledge) of the system and of other agents while learning and adapting their strategies during a simulation. The TEMMAS purpose is not to explicitly search for equilibrium points, but rather to reveal and assist to understand the complex and aggregate system behaviors that emerge from the interactions of the market agents.
We describe the structural MABS constituents by means of two concepts: i) the environmental entity, which owns a distinct existence in the real environment, e.g. a resource such as an electricity producer, or a decision-making agent such as a market bidder generator company, and ii) the environmental property, which is a measurable aspect of the real environment, e.g. the price of a bid or the demand for electricity. Hence, we define the environmental entity set, ET = { e1, . . . , en }, and the environmental property set, EY = { p1, . . . , pm }. The whole environment is the union of its entities and properties: E = ET ∪ EY .
The environmental entities, ET , are often clustered in different classes, or types, thus partitioning ET into a set, PET , of i , each containing entities that belong to disjoints subsets, PET the same class. Formally, PET = iPE1T , . . . , PEkT defines a full partition of ET , such that PET ⊆ ET and PET = i and PEiT j = ∅ ∀i 6= j. The partitioning ∪mia=y1.b..ek uPsEeTd to disting∩uisPhETbetween decision-making agents and available resources, e.g. a company that decides the biding strategy to pursue or a plant that provides the demanded power.
The environmental properties, EY , can also be clustered, in a similar way as for the environmental entities, thus grouping properties that are related. The partitioning may be used to express distinct categories, e.g. economical, electrical, ecological or social aspects. Another, more technical usage, is to separate constant parameters from dynamic state variables.
The factored state space representation. The state of the
simulated environment is implicitly defined by the state of all
its environmental entities and properties. We follow a factored
representation, that describes the state space as a set, V, of
discrete state variables [
From environmental entities to resources and agents. The
embodiment is central in describing the relation between the
entities and the environment [
The decision-making approach. Each agent perceives (the
market) and acts (sells or buys) and there are two main
approaches to develop the reasoning and decision-making
capabilities: i) the qualitative mental-state based reasoning,
such as the belief-desire-intention (BDI) architecture [
The qualitative mental-state approaches capture the relation between high level components (e.g. beliefs, desires, intentions) and tend to follow heuristic (or rule-based) decision-making strategies, thus being better fitted to tackle large-scale problems and worst fitted to deal with stochastic environments.
The quantitative decision-theoretic approaches deal with low level components (e.g., primitive actions and immediate rewards) and searches for long-term policies that maximize some utility function, thus being worst fitted to tackle large-scale problems and better fitted to deal with stochastic environments.
The electric power market is a stochastic environment and
we currently formulate medium-scale problems that can fit a
decision-theoretic agent model. Therefore, TEMMAS adaptive
agents (e.g., market bidders) follow a MDP based approach
and resort to experience (sampled sequences of states, actions
and rewards from simulated interaction) to search for optimal,
or near-optimal, policies using reinforcement learning methods
such as Q-learning [
III. TEMMAS DESIGN
Within the current design model of TEMMAS the electricity asset is traded through a spot market (no bilateral agreements), which is operated via a Pool institutional power entity. Each generator company, GenCo, submits (to Pool) how much energy, each of its generating unit, GenUnitGenCo, is willing to produce and at what price. Thus, we have: i) the power supply system comprises a set, EGenCo, of generator companies, ii) each generator company, GenCo, contains its own set, EGenUnitGenCo , of generating units, iii) each generating unit, GenUnitGenCo, of a GenCo, has constant marginal costs, and iv) the market operator, Pool, trades all the GenCos’ submitted energy.
The bidding procedure conforms to the so-called “block
bids” approach [
Therefore, we have: ET = { Pool } ∪ EGenCo ∪g∈EGenCo EGenUnitg where EY = { quantity, price, productionCost }. The quantity refers both to the supply and demand sides of the market. The price referes both to the supply bided values and to the market settled (by Pool) value.
The EGenCo contains the decision-making agents. The Pool is a reactive agent that always applies the same predefined auction rules in order to determine the market price and hence the block bids that clear the market. Each EGenUnitGenCo represents the GenCo’s set of available resources.
The resources’ specification. Each generating unit,
GenUnitGenCo, defines its marginal costs and constructs the
block bids according to the strategy indicated by its generator
company, GenCo. Each GenUnitGenCo calculates its marginal
costs according to, either the “WithHeatRate” [
The “WithHeatRate” formulation estimates the marginal cost, MC, by combining the variable operations and maintenance costs, vO&M, the number of heat rate intervals, nP at, each interval’s capacity, capi and the corresponding heat rate value, hri, and the price of the fuel, f P rice, being used; the marginal cost for a given i ∈ [1, nP at] interval is given by, (capi+1 × hri+1) − (capi × hri) blockCapi+1 ×f P rice MCi+1 = vO&M+ (1) where each block’s capacity is given by: blockCapi+1 = capi+1 − capi.
The “WithCO2” marginal cost, MC, combines the variable operations and maintenance costs, vO&M, the price of the fuel, f P rice, the CO2 cost, CO2cost, and the unit’s productivity, η, through the expression,
f P rice MC = × K + CO2cost + vO&M (2) η where K is a fuel-dependent constant factor, and CO2cost is given by,
CO2cost = CO2price ×
CO2emit η × K development platform. Figure 2 presents the general “agent’s perspective”, where the tasks and the goals are clustered into individual and social perspectives. Figure 3 gives additional detail on the construction of tasks and goals using INGENIAS. Market Operator (Pool) Buying Offers Market Results
Buyer individual perspective where CO2emit is the CO2 fuel’s emissions. Here all blocks have the same capacity; given a unit’s maximum capacity, maxCap, and a number of blocks, nBlocks, to sell, each block’s capacity is given by: blockCap = mnBaxloCckasp .
The decision-making strategies. Each generator company defines the bidding strategy for each of its generating units. We designed two types of strategies: a) the basic-adjustment , that chooses among a set of basic rigid options, and b) the heuristic-adjustment , that selects and follows a predefined well-known heuristic. There are several basic-adjustment strategies already defined in TEMMAS. Here we outline seven of those strategies, sttgi where i ∈ { 1, . . . , 7 }, available for a GenCo to apply: i) sttg1, bid according to the marginal production cost of each GenUnitGenCo (follow heat rate curves, e.g., cf. tables II and III), ii) sttg2, make a “small” increment in the prices of all the previous-day’s block bids, iii) sttg3, similar to sttg2, but makes a “large” increment, iv) sttg4, make a “small” decrement in the prices of all the previous-day’s block bids, v) sttg5, similar to sttg4, but makes a “large” decrement, vi) sttg6, hold the prices of all previous-day’s block bids, vii) sttg7 set the price to zero. There are two heuristic-adjustment defined strategies: a) the “Fixed Increment Price Probing” (FIPP) that uses a percentage to increment the price of last day’s transacted energy blocks and to decrement the non-transacted blocks, and b) “Physical Withholding based on System Reserve” (PWSR) that reduces the block’s capacity, as to decrement the next day’s estimated system reserve (difference between total capacity and total demand), and then bids the remaining energy at the maximum market price.
The agents’ decision process. The above strategies correspond to the GenCo agent’s primary actions. The GenCo has a set, EGenUnitGenCo , of generating units and, at each decision-epoch, it decides the strategy to apply to each generating unit, thus choosing a vector of strategies, s−−t→tg, where the ith vector’s component refers to the GenUnitGi enCo generating unit; thus, its action space is given by: A = ×|iE=G1enUnitGenCo | { sttg1, . . . , sttg7 }i ∪ { FIPP, PWSR }. The GenCo’s perceived market share, mShare, is used to characterize the agent internal memory so its state space is given by mShare ∈ [ 0..100 ]. Each GenCo is a MDP decision-making agent such that the decision process period represents a daily market. At each decision-epoch each agent computes its daily profit (that is regarded as an internal reward function) and the Pool agent receives all the GenCos’s block bids for the 24 daily hours and settles the hourly market price by matching offers in a classic supply and demand equilibrium price (we assume a hourly constant demand).
TEMMAS architecture and construction. The TEMMAS
agents along with the major inter-agent communication paths
are represented in the bottom region of Figure 1; the top
region represents the user interface that enables to specify the
each of the resources’ and agents’ configurable parameters.
The implementation of the TEMMAS architecture followed
the INGENIAS [
We used TEMMAS to build a specific electric market simulation model. We picked the inspiration from the Iberian Electricity Market (MIBEL – “Mercado Ibe´rico de Electricidade”) with Portuguese (e.g., EDP - “Electrividade de Portugal”, “Turboga´s”, “Tejo Energia”) and Spanish (e.g., “Endesa”, “Iberdrola”, “Union Fenosa”, “Hidro Canta´brico”, “Viesgo”, “Bas Natural”, “Elcoga´s”) generator companies. Regarding the total electricity capacity installed the Iberian market is composed of a major player (Spain) and a minor player (Portugal). Our experiments exploit the combined market behavior of a major and a minor electricity market players.
We abstracted intra-nation market details and modeled each country as a single generator company (with several generating units). Figure 4 uses INGENIAS notation to depict the hierarchical structure of the electricity market; the Pool (OMEL – “Operador do Mercado Ibe´rico de Electricidade”) settles the market price (and coupled bids) after the bids submitted by each GenCo (PT – “Portugal” and ES – “Spain”) according to a strategy that depends on the marginal production costs of each GenUnit.
We considered three types of generating units: i) one base load coal plant, CO, ii) one combined cycle plant, CC, to cover intermediate load, and iii) one gas turbine, GT, peaking unit. Table I shows the essential properties of each plant type and tables II and III shows the heat rate curves used to define the bidding blocks. The marginal cost was computed using expression ( 1 ); the bidding block’s quantity is the capacity increment, e.g. for CO, the 11.9 marginal cost bidding block’s quantity is 350 − 250 = 100 MW (cf. Table II, CO, top lines 2 and 1).
computed according to the respective GenUnits (cf. Table I). The “active” suffix (cf. Table IV, name column) means that the GenCo searches for its GenUnits best bidding strategies; i.e. “active” is a policy learning agent.
Our experiments have two main purposes: i) illustrate the TEMMAS functionality, and ii) analyze the agents’ resulting behavior, e.g. the learnt bidding policies, in light of the market specific dynamics.
We designed three experimental scenarios and Table IV shows the GenCo’s name along with its production capacity, Experiment #1. The experiment sets a constant, 600 MW, hourly demand for electricity. Figure 5 shows the GenCo active process of learning the bidding policy that gives the highest long-term profit. We used Q-learning, with an -greedy exploration strategy, which picks a random action with probability and behaves greedily otherwise (i.e., picks the action with the highest estimated action value); we defined = 0.2. The learning factor rate of Q-learning was defined as α = 0.01 and the discount factor (which measures the present value of future rewards) was set to γ = 0.5. Figure 6 shows the bid blocks that cleared the market (at the first hour of last simulated day). As there is no market competition the cheapest, CO, bids zero, the GT sets the market price (to its ceiling) and the most expensive 200 MW are distributed among the most expensive GenUnits (CC, GT). Therefore, the GenCo active agent found, for each perceived market share, mShare, the best strategy, s−−t→tg, to bid its GenUnits’ energy blocks.
Profit of GenCo _active 0 50
Base Coal (CO) 100 150 200 250 300 350 400 450 500 550 600
Capacity (MW)
Comb. Cycle (CC) Gas Turbine (GT) Fig. 6. The bid policy that maximizes profit (price ceiling is 180). [Exp. #1]
Experiment #2. The experiment sets a constant, 2000 MW, hourly demand for electricity. Figure 7 shows the market share evolution while GenCo minor&active learns to play in the market with GenCo major, which is a larger company with a fixed strategy: “bid each block 5 C higher than its marginal cost”. We see that GenCo minor&active gets around 18% (75 − 57) of market from GenCo major. To earn that market the GenCo minor&active learnt to lower its prices in order to exploit the “5 C space” offered by GenCo major fixed strategy.
Experiment #3. In this experiment both GenCos are “active”; the remaining is the same as in experiment #2. Figure 8 shows the market share oscillation while each company reacts to the other’s strategy to win the market. Despite the GenCos' Market Share
GenCo _major GenCo _minor&active GenCo_major&active GenCo_minor&active 0 10 20 30 40 50 60 70 80 90 100 Simulation Cycle (1 Day; 24 Hours) competition each company learns to secure its own fringe of the market.
GenCos' Market Share 0 500 1000 1500 2000 2500 3000 3500 4000 4500 5000 Simulation Cycle (1 Day; 24 Hours)
This paper describes our preliminary work in the construction of a MABS framework to analyze the macro-scale dynamics of the electric power market. Although both research fields (MABS and market simulation) achieved considerable progress there is a lack of cross-cutting approaches. We used the proposed MABS framework to support our preliminary work in the construction of the TEMMAS agent-based electricity market simulator.
Hence, our contribution is two folded: i) a comprehensive formulation of MABS, including the simulated environment and the inhabiting decision-making and learning agents, and ii) a simulation model (TEMMAS) of the electric power market framed in the proposed formulation.
Our initial results reveal an emerging and coherent market behavior, thus inciting us to further extend the experimental setup with additional bidding strategies and to incorporate specific market rules, such as congestion management and pricing regulation mechanisms.