This paper describes a multi-agent based simulation (MABS) framework to construct an artificial electric power market populated with learning agents. The artificial market, named TEMMAS (The Electricity Market Multi-Agent Simulator), explores the integration of two design constructs: i) the specification of the environmental physical market properties, and ii) the specification of the decision-making (deliberative) and reactive agents. TEMMAS is materialized in an experimental setup involving distinct power generator companies which operate in the market and search for the trading strategies that best exploit their generating units' resources. The experimental results show a coherent market behavior that emerges from the overall simulated environment.
The start-up of nation-wide electric markets, along with its recent expansion to intercountry markets, aims at providing competitive electricity service to consumers. The new market-based power industry calls for human decision-making in order to settle the energy assets' trading strategies. The interactions and influences among the market participants are usually described by game theoretic approaches which are based on the determination of equilibrium points to which compare the actual market performance [1], [2]. However, those approaches find it difficult to incorporate the ability of market participants to repeatedly probe markets and adapt their strategies. Usually, the problem of finding the equilibria strategies is relaxed (simplified) both in terms of: i) the human agents' bidding policies, and ii) the technical and economical operation of the power system.
As an alternative to the equilibrium approaches, the multi--agent based simulation (MABS) comes forth as being particulary well fitted to analyze dynamic and adaptive systems with complex interactions among constituents [3], [4].
In this paper we describe a MABS modeling framework that provides constructs for the (human) designer to specify a dynamic environment, its resources, observable properties and its inhabitant decision-making agents. We used the framework to capture the behavior of the electricity market and to build a simulator, named TEMMAS (The Electricity Market Multi-Agent Simulator), which incorporates the operation of several generator company (GenCo) operators, each with distinct power generating units (GenUnit), and a market operator (Pool) which computes the hourly market price (driven by the electricity demand).
TEMMAS agents exhibit bounded rationality, i.e., they make decisions based on local information (partial knowledge) of the system and of other agents while learning and adapting their strategies during a simulation. The TEMMAS purpose is not to explicitly search for equilibrium points, but rather to reveal and assist to understand the complex and aggregate system behaviors that emerge from the interactions of the market agents.
We describe the structural MABS constituents by means of two concepts: i) the environmental entity, which owns a distinct existence in the real environment, e.g. a resource such as an electricity producer, or a decision-making agent such as a market bidder generator company, and ii) the environmental property, which is a measurable aspect of the real environment, e.g. the price of a bid or the demand for electricity. Hence, we define the environmental entity set, E T = { e 1 , . . . , e n }, and the environmental property set, E Y = { p 1 , . . . , p m }. The whole environment is the union of its entities and properties:
The environmental entities, E T , are often clustered in different classes, or types, thus partitioning E T into a set, P E T , of disjoints subsets, P i E T , each containing entities that belong to the same class. Formally,
The partitioning may be used to distinguish between decision-making agents and available resources, e.g. a company that decides the biding strategy to pursue or a plant that provides the demanded power.
The environmental properties, E Y , can also be clustered, in a similar way as for the environmental entities, thus grouping properties that are related. The partitioning may be used to express distinct categories, e.g. economical, electrical, ecological or social aspects. Another, more technical usage, is to separate constant parameters from dynamic state variables.
The factored state space representation. The state of the simulated environment is implicitly defined by the state of all its environmental entities and properties. We follow a factored representation, that describes the state space as a set, V, of discrete state variables [5]. Each state variable, v i ∈ V, takes on values in its domain D( v i ) and the global (i.e., over E) state space, S ⊆ × vi∈V D( v i ), is a subset of the Cartesian product of the state variable domains. A state s ∈ S is an assignment of values to the set of state variables V. We define f C , C ⊆ V, as a projection such that if s is an assignment to V, f C ( s ) is the assignment of s to C; we define a context c as an assignment to the subset C ⊆ V; the initial state variables of each entity and property are defined, respectively, by the functions init E T :
From environmental entities to resources and agents. The embodiment is central in describing the relation between the entities and the environment [6]. Each environmental entity can be seen as a body, possibly with the capability to influence the environmental properties. Based on this idea of embodiment, two higher-level concepts (decoupled from the environment, E, characterization) are introduced: i) agent, owing reasoning and decision-making capabilities, and ii) resource, without any reasoning capability. Thus, given a set of agents, Υ, we define an association function embody : Υ → E T , which connects an agent to its physical entity. In a similar way, given a set of resources, Φ, we define the mapping function identity : Φ → E Y . We consider that |E| = |Υ| + |Φ|, thus each entity is either mapped to an agent or to a resource; there is no third category.
The decision-making approach. Each agent perceives (the market) and acts (sells or buys) and there are two main approaches to develop the reasoning and decision-making capabilities: i) the qualitative mental-state based reasoning, such as the belief-desire-intention (BDI) architecture [7], which is founded on logic theories, and ii) the quantitative, decision-theoretic, evaluation of causal effects, such as the Markov decision process (MDP) support for sequential decision-making in stochastic environments. There are also hybrid approaches that combine the qualitative and quantitative formulations [8], [9].
The qualitative mental-state approaches capture the relation between high level components (e.g. beliefs, desires, intentions) and tend to follow heuristic (or rule-based) decision--making strategies, thus being better fitted to tackle large-scale problems and worst fitted to deal with stochastic environments.
The quantitative decision-theoretic approaches deal with low level components (e.g., primitive actions and immediate rewards) and searches for long-term policies that maximize some utility function, thus being worst fitted to tackle large-scale problems and better fitted to deal with stochastic environments.
The electric power market is a stochastic environment and we currently formulate medium-scale problems that can fit a decision-theoretic agent model. Therefore, TEMMAS adaptive agents (e.g., market bidders) follow a MDP based approach and resort to experience (sampled sequences of states, actions and rewards from simulated interaction) to search for optimal, or near-optimal, policies using reinforcement learning methods such as Q-learning [10] or SARSA [11].
Within the current design model of TEMMAS the electricity asset is traded through a spot market (no bilateral agreements), which is operated via a Pool institutional power entity. Each generator company, GenCo, submits (to Pool) how much energy, each of its generating unit, GenUnit GenCo , is willing to produce and at what price. Thus, we have: i) the power supply system comprises a set, E GenCo , of generator companies, ii) each generator company, GenCo, contains its own set, E GenUnitGenCo , of generating units, iii) each generating unit, GenUnit GenCo , of a GenCo, has constant marginal costs, and iv) the market operator, Pool, trades all the GenCos' submitted energy.
The bidding procedure conforms to the so-called "block bids" approach [12], where a block represents a quantity of energy being bided for a certain price; also, GenCos are not allowed to bid higher than a predefined price ceiling. Thus, the market supply essential measurable aspects are the energy price, quantity and production cost. The consumer side of the market is mainly described by the quantity of demanded energy; we assume that there is no price elasticity of demand (i.e., no demand-side market bidding).
Therefore, we have:
The quantity refers both to the supply and demand sides of the market. The price referes both to the supply bided values and to the market settled (by Pool) value.
The E GenCo contains the decision-making agents. The Pool is a reactive agent that always applies the same predefined auction rules in order to determine the market price and hence the block bids that clear the market. Each E GenUnitGenCo represents the GenCo's set of available resources.
The resources' specification. Each generating unit, GenUnit GenCo , defines its marginal costs and constructs the block bids according to the strategy indicated by its generator company, GenCo. Each GenUnit GenCo calculates its marginal costs according to, either the "WithHeatRate" [13]) or the "WithCO 2 " [14] formulation.
The "WithHeatRate" formulation estimates the marginal cost, MC, by combining the variable operations and maintenance costs, vO&M, the number of heat rate intervals, nP at, each interval's capacity, cap i and the corresponding heat rate value, hr i , and the price of the fuel, f P rice, being used; the marginal cost for a given i ∈ [1, nP at] interval is given by,
) where each block's capacity is given by: blockCap i+1 = cap i+1 − cap i .
The "WithCO 2 " marginal cost, MC, combines the variable operations and maintenance costs, vO&M, the price of the fuel, f P rice, the CO 2 cost, CO 2 cost, and the unit's productivity, η, through the expression,
where K is a fuel-dependent constant factor, and CO 2 cost is given by,
where CO 2 emit is the CO 2 fuel's emissions. Here all blocks have the same capacity; given a unit's maximum capacity, maxCap, and a number of blocks, nBlocks, to sell, each block's capacity is given by: blockCap = maxCap nBlocks . The decision-making strategies. Each generator company defines the bidding strategy for each of its generating units. We designed two types of strategies: a) the basic-adjustment, that chooses among a set of basic rigid options, and b) the heuristic-adjustment, that selects and follows a predefined well-known heuristic. There are several basic-adjustment strategies already defined in TEMMAS. Here we outline seven of those strategies, sttg i where i ∈ { 1, . . . , 7 }, available for a GenCo to apply: i) sttg 1 , bid according to the marginal production cost of each GenUnit GenCo (follow heat rate curves, e.g., cf. tables II and III), ii) sttg 2 , make a "small" increment in the prices of all the previous-day's block bids, iii) sttg 3 , similar to sttg 2 , but makes a "large" increment, iv) sttg 4 , make a "small" decrement in the prices of all the previous-day's block bids, v) sttg 5 , similar to sttg 4 , but makes a "large" decrement, vi) sttg 6 , hold the prices of all previous-day's block bids, vii) sttg 7 set the price to zero. There are two heuristic-adjustment defined strategies: a) the "Fixed Increment Price Probing" (FIPP) that uses a percentage to increment the price of last day's transacted energy blocks and to decrement the non-transacted blocks, and b) "Physical Withholding based on System Reserve" (PWSR) that reduces the block's capacity, as to decrement the next day's estimated system reserve (difference between total capacity and total demand), and then bids the remaining energy at the maximum market price.
The agents' decision process. The above strategies correspond to the GenCo agent's primary actions. The GenCo has a set, E GenUnitGenCo , of generating units and, at each decision-epoch, it decides the strategy to apply to each generating unit, thus choosing a vector of strategies, − − → sttg, where the i th vector's component refers to the GenUnit i GenCo generating unit; thus, its action space is given by: A = × |EGenUnit GenCo | i=1 { sttg 1 , . . . , sttg 7 } i ∪ { FIPP, PWSR }. The GenCo's perceived market share, mShare, is used to characterize the agent internal memory so its state space is given by mShare ∈ [ 0..100 ]. Each GenCo is a MDP decision-making agent such that the decision process period represents a daily market. At each decision-epoch each agent computes its daily profit (that is regarded as an internal reward function) and the Pool agent receives all the GenCos's block bids for the 24 daily hours and settles the hourly market price by matching offers in a classic supply and demand equilibrium price (we assume a hourly constant demand).
TEMMAS architecture and construction. The TEMMAS agents along with the major inter-agent communication paths are represented in the bottom region of Figure 1; the top region represents the user interface that enables to specify the each of the resources' and agents' configurable parameters. The implementation of the TEMMAS architecture followed the INGENIAS [15] methodology and used its supporting development platform. Figure 2 presents the general "agent's perspective", where the tasks and the goals are clustered into individual and social perspectives. Figure 3 gives additional detail on the construction of tasks and goals using INGENIAS.
We used TEMMAS to build a specific electric market simulation model. We picked the inspiration from the Iberian Electricity Market (MIBEL -"Mercado Ibérico de Electricidade") with Portuguese (e.g., EDP -"Electrividade de Portugal", "Turbogás", "Tejo Energia") and Spanish (e.g., "Endesa", "Iberdrola", "Union Fenosa", "Hidro Cantábrico", "Viesgo", "Bas Natural", "Elcogás") generator companies. Regarding the total electricity capacity installed the Iberian market is composed of a major player (Spain) and a minor player (Portugal). Our experiments exploit the combined market behavior of a major and a minor electricity market players.
We abstracted intra-nation market details and modeled each country as a single generator company (with several generating units). Figure 4 uses INGENIAS notation to depict the hierarchical structure of the electricity market; the Pool (OMEL -"Operador do Mercado Ibérico de Electricidade") settles the market price (and coupled bids) after the bids submitted by each GenCo (PT -"Portugal" and ES -"Spain") according to a strategy that depends on the marginal production costs of each GenUnit. We considered three types of generating units: i) one base load coal plant, CO, ii) one combined cycle plant, CC, to cover intermediate load, and iii) one gas turbine, GT, peaking unit. Table I shows the essential properties of each plant type and tables II and III shows the heat rate curves used to define the bidding blocks. The marginal cost was computed using expression ( 1 ); the bidding block's quantity is the capacity increment, e.g. for CO, the 11.9 marginal cost bidding block's quantity is 350 − 250 = 100 MW (cf. Table II, CO, top lines 2 and 1).
Our experiments have two main purposes: i) illustrate the TEMMAS functionality, and ii) analyze the agents' resulting behavior, e.g. the learnt bidding policies, in light of the market specific dynamics.
We designed three experimental scenarios and Table IV shows the GenCo's name along with its production capacity, Experiment #1. The experiment sets a constant, 600 MW, hourly demand for electricity. Figure 5 shows the GenCo active process of learning the bidding policy that gives the highest long-term profit. We used Q-learning, with an -greedy exploration strategy, which picks a random action with probability and behaves greedily otherwise (i.e., picks the action with the highest estimated action value); we defined = 0.2. The learning factor rate of Q-learning was defined as α = 0.01 and the discount factor (which measures the present value of future rewards) was set to γ = 0.5. Figure 6 shows the bid blocks that cleared the market (at the first hour of last simulated day). As there is no market competition the cheapest, CO, bids zero, the GT sets the market price (to its ceiling) and the most expensive 200 MW are distributed among the most expensive GenUnits (CC, GT). Therefore, the GenCo active agent found, for each perceived market share, mShare, the best strategy, − − → sttg, to bid its GenUnits' energy blocks. Experiment #2. The experiment sets a constant, 2000 MW, hourly demand for electricity. Figure 7 shows the market share evolution while GenCo minor&active learns to play in the market with GenCo major, which is a larger company with a fixed strategy: "bid each block 5C higher than its marginal cost". We see that GenCo minor&active gets around 18% (75 − 57) of market from GenCo major. To earn that market the GenCo minor&active learnt to lower its prices in order to exploit the "5C space" offered by GenCo major fixed strategy.
Experiment #3. In this experiment both GenCos are "active"; the remaining is the same as in experiment #2. Figure 8 shows the market share oscillation while each company reacts to the other's strategy to win the market. Despite the competition each company learns to secure its own fringe of the market.
This paper describes our preliminary work in the construction of a MABS framework to analyze the macro-scale dynamics of the electric power market. Although both research fields (MABS and market simulation) achieved considerable progress there is a lack of cross-cutting approaches. We used the proposed MABS framework to support our preliminary work in the construction of the TEMMAS agent-based electricity market simulator.
Hence, our contribution is two folded: i) a comprehensive formulation of MABS, including the simulated environment and the inhabiting decision-making and learning agents, and ii) a simulation model (TEMMAS) of the electric power market framed in the proposed formulation.
Our initial results reveal an emerging and coherent market behavior, thus inciting us to further extend the experimental setup with additional bidding strategies and to incorporate specific market rules, such as congestion management and pricing regulation mechanisms.