-Thinking about the complexity of the data center environment, this paper explores the characteristics of knowledgebased agents to discover energy savings opportunities within these dynamic systems. The goal is to concentrate on the most significant problems while, at the same time, create some flexibility in which undesired metrics are acceptable in face of their positive and negative impact analysis from an energy perspective.
Over the last years, managing the energy efficiency of
ICT (Information and Communication Technology) has
dramatically emerged as one of the most critical environmental
challenges to be dealt with. Energy consumption and
energy efficiency of ICT centers became priority due to high
computing demand and new environmental regulations. Cloud
computing paradigm has been an important contributor for
increasing the data centers importance, where users are always
connected through different devices. The Greenpeace 2012
year report [
In the present work, we consider the requirements for enabling energy-efficient mechanisms in service-based systems, based on a goal-event-action adaptive approach. One of the problems that emerges when trying to apply adaptive mechanisms based on monitoring in this area is that we need to select adaptation actions in an uncertain environment, in which event occurrences and actions to achieve goals depend on the context of execution.
Looking to provide an integrated solution for reasoning
about such complex environment, this paper focuses on the
data center energy aspect. Mechanisms are described to: (I)
identify potential energy saving without compromising
performance metrics and (II) reason about the best action to
be taken in order to achieve desired levels of satisfaction.
The solution is mainly composed by two knowledge-based
agents [
This paper is organized as following: Section two introduces metrics, called indicators, for controlling performance and
Copyright c 2013 for the individual papers by the papers’ authors. Copying permitted only for private and academic purposes. This volume is published and copyrighted by its editors. energy related issues of an data center; section three discusses about the problem of identifying system threats when we are dealing with many and heterogeneous variables; the proposed solution, an Integrated Energy-aware Framework (IEF), is described in section four; and finally section five briefly discusses some remarking points.
IT equipment is the data center’s core and it is composed basically by servers, network equipment, and storage devices that are available for service-based applications (SBAs) in a loosely coupled manner. They are monitored by several sensors installed throughout the facility, which provide information both from physical (e.g., server energy consumption) and logical layers (e.g., web-service response time). Such data represent levels of satisfaction when we use them to calculate performance and energy indicators. They are defined in terms of thresholds, which can trigger alarming or warning alerts when their limits are not respected. When it happens, repairing actions are selected and enacted in order to restore desired levels of satisfaction.
Such elasticity is only possible when there is a comprehensive view of the system, in which the different elements relationships are stipulated and alerts are properly signed as system threats. In this section we describe these indicators and how they can be used to identify system threats. An indicator is defined as a metric that provides information about the status of the underlying system and is created according to the meta-model described in Figure 1. The abstract class Indicator defines both quality and energy related indicators which are identified by the attribute type. The attribute importance is based on the user’s preferences and it dictates the indicator priority in case of multiple violations.
The indicator violation is defined by the Threshold class, which is composed by two lower-bounds, minAlarm (amin) and minWarning (wmin), and two higher-bounds, maxAlarm (amax) and maxWarning (wmax). Indicators alarming thresholds represent critical boundaries that should not be violated in order to keep the system soundness. On the other hand, warning thresholds are less critical as their violation can be accepted (although not desired). Thus, warning thresholds can be seem as the system elasticity, which are used to keep the majority of the indicators outside the alarming zones. An indicator is not violated, i.e. green, when its current value is within both warning and alarming boundaries. Otherwise, we can have warning or alarming violation. In the first case, yellow, indicator values are between the warning and alarming thresholds. In the second case, red, the indicator value violates both warning and alarming thresholds. It shows an unacceptable situation of one or more system components. Alarming indicators have priority to be solved with respect to warning ones as they can cause higher damage to the system. The current indicator situation (green, yellow or red) is represented by the attribute status.
The indicator last attribute, acceptance, defines the accepted interval in which an indicator can stay in a violated state (warning or alarming violation) without requiring adaptation. This attribute is important in order to avoid unnecessary adaptation enactment, where the violation is temporary and might not require adaptation. For instance, let us suppose that the action VM migration may violate the indicator availability of an application. In this example, the acceptance attribute shall dictate the maximum time interval allowed for which the indicator availability can stay violated without triggering a new adaptation action, i.e., the maximum amount of time the VM migration action can take without causing a new side-effect.
The indicator value is obtained through the indicator formula calculation, which uses the information provided by the monitoring system, which is composed by several sensors. Indicators are divided in basic and composed. An indicator is called as basic (BasicIndicator class) when its calculation formula is either a direct measure from the monitoring system variable or a combination of several monitoring variables within a formula. Instead, composed indicators (ComposedIndicator class) formulae use other basic indicators as input. They do not use other composed indicators in order to avoid infinity recursive compositions.
Fig. 1. Indicators meta-model.
Architectural layer
1 StoredValues * *
Indicator 0..* +Name +Type +Importance 1 +Status +Acceptance 1
Thresholds +minAlarm 1..* +minWarning +maxWarning +maxAlarm toringMetric 1
Basic Indicator
1..* 1 +Formula
Composed Indicator * +CompositionMetric
of SBAs, energy consumption can be constantly monitored
by specific indicators called Green Performance Indicators
(GPIs) [
III. THE PROBLEM OF THREATS IDENTIFICATION Once indicators are defined and their value calculated, the system shall be able to recognize which ones represent system threats, i.e., which ones can harm the system. In this way, an isolated indicator violation does not represent necessarily a system threat. The identification of system threats introduces new complexity boundaries to the approach, which need to be selective in mining the monitored data in order to identify relevant information to support the decision making mechanisms.
These threats are identified through timed event occurrences, which can appear within four different scenarios. In these scenarios, an event occurrence is associated with the execution of adaptation actions or indicators violations, which occur within defined time windows. The scenarios are: I - The adaptation action is not effective and event occurrences are created following the same pattern.
II - The adaptation action is temporary effective, solving the problem during a short time interval which defines the action approval period. In this scenario, the same event occurrence comes back within the action approval period which means that the enacted action was not suitable to eliminate the threat and a different solution should be taken.
III - The adaptation action is fully effective to solve the identified threat. However, its execution generates non expected side-effects, creating a completely new threat. This scenario is quite dangerous as it may create an infinity cycle of action enactment and threat creation. IV - The adaptation action seems to be fully effective, but the same threat appears again after some time. Differently from scenario II, in this scenario the time gap is quite long and the threat appears in this first event occurrence.
In order to make clear the difference between the last three scenarios, let us consider a flat tire example. After the identification of the problem, one possible action to solve the problem is filling the tire with air. If the reason of the problem is a tire hole, this action is not effective since the tire will be flat again in few minutes or hours. This situation is represented in Scenario II. Instead, if another action is chosen, such as replace the tire, different events can be raised due to the enactment of the action. In the example, the events car unstable (due to inadequate spare tire) or drive not safe (due to the lack of available spare tire within the car) can be raised. This situation is represented in Scenario III. Even if the tire replacement effectively solves the problem, it does not prevent that a new hole appears in the future (Scenario IV). A more detailed analysis with respect to an adaptation action feedback and the implications of failed actions is described in the following section.
IV. INTEGRATED ENERGY-AWARE FRAMEWORK As described in the previous section, indicators violation may represent system threats that have to be eliminated for the system soundness. The threat elimination is represented by the enactment of combined adaptation actions that bring positive and negative impact to the overall system. In order to identify these system threats and to select the best set of adaptation actions to be executed, we use a knowledge-based agent approach. In this section we detail all the elements present in an Integrated Energy-aware Framework (IEF), which are responsible for identifying and eliminating threats that may compromise the system.
Figure 2 shows the main elements that compose the proposed framework. First, the runtime and monitoring system provides raw data about the underlying environment. This information is used to calculate the defined indicators by the indicator calculation module. The first agent, Agent 1, is responsible to verify both monitored variables and indicators values in order to recognize possible situations that represent a threat to the overall system. If a threat is identified, the agent creates several event occurrences that are analyzed by the second agent, Agent 2, looking to separate the different types of events and, in particular, the ones that require the enactment of adaptation actions (meaningful events). To do so, both agents inference rules are based on the system knowledge base, which contains current monitored data and all past experiences. As new information is arriving, the knowledge base is constantly updated by both inference engines. These changes impact on both the conceptualization of system threats and the selection of adaptation actions.
Finally, the adaptation parameterization module is responsible to approve or to disapprove the selected adaptation actions by the system manager and, if approved, he should provide the necessary actions parameters values.
Knowledge
base Inference engine 1 Agent 1
Prior knowledge
Agent 2 Inference engine 2 Indicators values
Adapta on set
Adapta on parameteriza on System Manager
Supervise
Enable Indicator calcula on Monitoring data
Monitoring data
Run me and monitoring
Based on the monitored data and the indicator calculated values, Agent 1 recognizes three different levels of threats, which might or might not result in the creation of an event occurrence. An event occurrence represents an event that is currently harming one or more indicators fulfillment and, therefore, adaptation actions shall be enacted in order to restore desired levels of satisfaction. The creation of an event occurrence is based on the underlying environment, which is represented by the instantiation of some context rules.
Due to the spread usage of the word ‘context’ and to avoid
misunderstandings, we adopt the context definition provided
by [
This agent PEAS description is:
Performance: Number of warning and alarming indicators violation; Environment: Indicators levels identified as yellow, orange and red; Actuators: Identification of three levels of system threats and creation of event occurrences;
Sensors: Indicators value and thresholds.
1) Threats identification: Considering that data is
continuously produced by the monitoring system, the agent is
responsible to recognize situations that might threaten the
indicators fulfillment. In order to deal with such a huge
bunch of data rapidly, we adopt Stream Reasoning techniques
proposed by [
Differently from other types of data, streams are consumed by queries registered into a stream processor, that continuously produce answers. A common and important simplification applied by stream engines is that they process information within windows, defined as periods of time slots in which the flowing information should be considered. Such windows are continuously evolving due to the arrival of new data in the stream, whereas data falling outside of the windows is lost, in other words, it expires. The window size defines the stream expiration time which represents the triple arrival timestamp plus the window size, assuming the window size to be constant (time-invariant). We also assume time as a discrete and linear ordered variable parameter. Therefore, if the window is defined as 3 time slots long and a time slot is 5 seconds long, a triple entering at time will expire at time + 15s. The expiration time of derived triples depends on the minimum expiration time of the triples it was derived from.
Considering the current materialized predicates window, the
Reaisvede evnentts identification module derives different levels
of threat in order to raise an event based on the a set of rules. A
threat is identified if one or more indicators are violated with
1Resource Description Framework (RDF) is a W3C recommendation for
resource description [
The identification of an indicator violation is represented by the creation of the following entailment:
Status(ih; ‘yellow’)
Status(ih; ‘red’) ! ih.value 2 Yh=
T Yh = [ ([ahmin; ahmax](t) n [whmin; whmax](t))
t=1 ! ih.value 2 Rh=
T Rh = Uh n [ [ahmin; ahmax](t) t=1 (1) where ih.value is the calculated value of indicator ih and Uh is the set of all possible values the indicator can assume and, therefore, vh 2 Uh. The set of values that does violate the indicator’s warning thresholds (max or min) is defined as Yh. In the same manner, Rh represents the set of values that violates alarming (max or min) thresholds, in which (Yh [ Rh) Uh. Finally, T represents thresholds sets of ih where t defines the thresholds dimensions fahmin; whmin; vh; whmax; ahmaxg such that ahmin whmin whmax ahmax.
The second level of a threat tries to identify warned violated indicators (yellow) that are likely to become alarmed violated indicators (red). We represent these indicators as orange ones. The identification of an orange indicator is similar to the identification of yellow ones, but narrowing alarming thresholds. Let us consider that Wh represents the set of triples of ih such that warning threshold (min or max) is violated. The alarming thresholds are narrowed based on Wh standard deviation (Rh0). The calculation of (Wh) is s 1 PN (xn )2 where N n=1 N is the number of triples in Wh. As we aim only the triples that violate the narrowed alarming thresholds but do not violate the normal alarming thresholds, we have Rh0 \ Yh. Thus, an indicator status is defined as orange according to the following:
Status(ih; ‘orange’) ! ih.value 2 Rh0 \ Yh=
T Rh0 = Uh n [ [ahmin + (Wh); ahmax t=1 (Wh)](t) (2)
Finally, the third level of threat actually raises an event after we have identified indicators as red or orange. However, differently from the first level of threat, the violation lasts more than it is accepted by the system. The indicator alarming violation is linked with an acceptance value acph, measured in number of monitored time slots the indicator can stay violated without the enactment of adaptation action. As each triple is associated with a timestamp, the calculation of the (3) violation duration is obtained by subtracting the timestamp of the last triple by the first one. Considering that Vh is the set of indicators with either red or orange violation, it is calculated the violation duration by M axT ime(Vh) M inT ime(Vh), where M axT ime() returns the last violated triple window and M inT ime() returns the first violated triple window. Thus, an event occurrence is created when the following expression holds:
M axT ime(Vh)
M inT ime(Vh) > acph= 8v 2 Vh : Status(v; ‘red’) _ Status(v; ‘orange’)
event occurrence eci;j 2 ECi means that event ei 2 E is raised, where ECi is the set of occurrences of event ei. Event occurrences are created by the event occurrence module. They are defined by the following attributes: h timetsamp, value, window, direction, significance, severity i, where: Timestamp: it is the timestamp of the RDF triple that triggers eci;j creation.
Value: it is the monitored variable value of the RDF triple that triggers eci;j creation.
Window: it represents the window in which the RDF triple that triggers eci;j is placed.
Direction: it dictates if the obtained value is increasing or decreasing based on threshold violation, i.e., max or min. This information is important in order to support the adaptation action selection phase.
Significance: it is a value between [0::1] resulted from two variables: variable significance and normalized variable value based on the indicator thresholds. The significance is important (together with impact) to define which event occurrences have priority to be eliminated by the adaptation actions.
Severity: this attribute defines quantitatively the severity level of event occurrence eci;j over one or more goals.
Algorithm 1 details the process of creating a new event occurrence eci;j . The algorithm input parameter is the RDF triple tk that satisfies Eq.3, i.e., an indicator alarming violation with longer duration than it is accepted.
Require: tk 1: ih getInd(tk) 2: W0 getVtriples(tk; ih) 3: gp getGoal(ih) 4: E0 8ei 2 E : ei impac!t gp 5: for all ei 2 E0 do 6: emci emci:event=ei (ECM ) 7: for all neg ef f ectj 2 emci:negEf f ect do 8: ctx getContextValues(neg ef f ectj ) 9: if checkCondRule(ctx,neg ef f ectj ) then 10: eci;j new EC(ei; ih; neg ef f ectj ) 11: end if 12: end for 13: end for
The first step is to find out the set of triples that represent red or orange violations, according to Eq. 3. This is done by function getVtriples() (line 2), which retrieves the set of triples W 0 = hih ind:value vhi. Goals (G) are defined as indicators thresholds and events (E) in terms of monitored variables. If the calculation of an indicator value depends on more than one variable, it means that one goal is impacted by more than one event. Thus, we need to identify the event ei 2 E0, where E0 E represents the set of candidates events to be raised due to ih violation. In order to create E0 we need to identify the goal gp that represents the violated indicator threshold ih through the function getGoal() (line 3). Based on gp we select all events that hold impact relationship like ei ! gp (line 4). The identification the right event that is triggering the indicator violation is based on the event context-model and, therefore, we need to select the event context-models with respect the event candidates E0 (lines 56). An event context-model emci may have several positive and negative effects context (line 7). Therefore, an event is raised when negative conditional context rules hold (line 9-10), which are based on current context variable values (line 8). Note that an indicator violation can raise more than one event and, each event can create one or more event occurrences, depending on the defined event context conditional rules.
Whenever event occurrences are identified and raised by Agent 1, the system shall be able to recognize events relationships and properly enact corrective actions. This is done by Agent 2, which analyzes situations when one event is caused by another through the enactment of its related adaptation action. By knowing these relationships, the agent drives the system adaptive behavior.
Agent 2 PEAS description is:
Performance: Number of raised event occurrences; Environment: Current and previous materialized windows; Actuators: Analysis of triggered adaptation actions (AdaptationRequired, OngoingAdaptation, Adaptation
fective) and selection of new ones; Sensors: Event occurrences, event status and enacted actions within materialized windows.
1) Event analysis: The existence of raised events through event occurrences indicates that adaptation actions have to be enacted. However, we argue the approach should be able to identify the different raised events in order to support the adaptation action selection.
There are five different status transitions which the relation status can represent: (i) Ready, eci;j has no relation with past ones; (ii) new, an adaptation was not yet selected; (iii) wip (work in progress), there are actions under execution; (iv) done, all actions are set as ‘finished’; and (v) renew, new eci;j regarding to the same event during the approval period.
The detailed description of the rules used in the states transitions are expressed as follows:
\AdaptationRequired00 r1a : W+(Status(ei; ‘new’)) :- Wbefore(Status(ei; ‘ready’)); Wins(Occur(ei; eci;j ) \AdaptationStillRequired00 r2a : W+(Status(ei; ‘renew’)) :- Wbefore(Status(ei; ‘done’)); Wins(Occur(ei; eci;j ));
W(Enact(av; ‘finished’)) \OngoingAdaptation00 r3a : W+(Status(ei; ‘wip’)) :- Wbefore(Status(ei; ‘new’));
Wins(Occur(ei; eci;j )); :W(Enact(av; ‘finished’)) r3b : W+(Status(ei; ‘wip’)) :- Wbefore(Status(ei; ‘wip’));
Wins(Occur(ei; eci;j )); :W(Enact(av; ‘finished’)) r3c : W+(Status(ei; ‘wip’)) :- Wbefore(Status(ei; ‘renew’)); Wins(Occur(ei; eci;j )); :W(Enact(av; ‘finished’)) \AdaptationCompleted00 r4a : W+(Status(ei; ‘done’)) :- Wbefore(Status(ei; ‘wip’)); :Wins(Occur(ei; eci;j ));
W(Enact(av; ‘finished’)) r4b : W+(DoneAt(ei; ) :- W+ (Status(ei; ‘done’)); ? \AdaptationEf f ective00 r5a : W+(Status(ei; ‘ready’)) :- Wbefore(Status(ei; ‘done’)); W(DoneAt(ei; ?doneT ime); (Approval(ei; ?approvalP eriod)); now() ?doneT ime >?approvalP eriod where ?time represents the current timestamp. In addition, the rules are based on the following materialized windows: W is current materialized window, W + contains the derived triples to be added in W , W bef ore represents the previous materialized window, and W in represents the new triples that are coming from the event identification module, i.e., event occurrences or monitored variables values. Adaptation actions are required only for raised events that hold either ‘new’ or ‘renew’ status.
2) Adaptation selection: When there are event occurrences eci;j identified as “new” or “renew” by the relation Status, the adaptation selection is triggered. Actions are combined in order to create an adaptation strategy, which is a composed by a set of coordinated adaptation actions, i.e., actions that are executed in sequence and/or parallel (coordination). It can happen that this set is composed by only one action, so it does not require any coordination.
An adaptation action is described by the following attributes: Type identifies if the action consequence is regarding functionality/quality reduction or resource reallocation; Duration is a time interval attribute that specifies the expected time interval, in terms of maximum and minimum, that the action takes to complete its execution; Cost is an interval attribute that represents the expected cost of the action execution and might depend on the action parameters defined by the system manager in the adaptation parameterization module; ApprovalPeriod is the expected time interval defined to validate the action effectiveness; AvailabilityCond represents the set of conditional rules that should be satisfied in order to enable the action execution; TriggerCond also represents a set of conditional rules, but it describes triggering conditions that can be either reactive or proactive; Group identifies the action group from an energy perspective; and finally Action is the adaptation action implementation, i.e., what the action should do.
Considering these attributes, Agent 2 selects the most suitable set of adaptation actions looking to eliminate event occurrences that are causing indicators violation. Five steps are used to do so. The first step (Step 1) identifies the incoming event occurrences that did not trigger adaptation yet. This identification is based on the event status attribute.
The next step (Step 2) aims to cut off the list of existing adaptation actions in order to keep only the available and suitable ones. In the first case, available actions, we use the action’s availability conditional rules. These rules are connected to single BPs that were previously designed to execute the action. For instance, the adaptation action skip task can be enacted if and only if the task is defined as optional task, i.e., the task is not critical. In the second case, suitable actions, we use the actions triggering conditions attribute in order to rule out actions from the list that were not designed to stop the incoming event occurrence. Each adaptation action is associated with events (event-trigger) or indicators violation (indicator-trigger) and, depending on the event occurrence, we keep only the actions that are directly or indirectly related to each other through an indicator violation, which is represented by contribution relations within the goalbased model.
Having the subset of actions provided by Step 2, the next
step (Step 3) determines the optimal set of actions that are
supposed to stop the arrival of new event occurrence with
minimal side-effects. This step can be divided into four
substeps, which are:
i An indicator violation may represent a violation of
other indicators that compose the first one. Thus,
the aim of this sub-step is to find the
indicatorsbase that have been violated. This is done through
and/or decomposition relationships among
indicators within the goal-based model;
ii Search for previous situations in which the current threat
was identified and, most important, what adaptation
strategies were selected with their obtained results;
iii A verification process estimates each action effects, both
positive and negative. The actions that propagate more
negative effects than positive or do not satisfy minimal
duration time or cost are ruled out of the candidate set.
After we ensure that all quality and energy constraints
are satisfied, Constraint Satisfaction Optimization
Problem is used;
iv Finally, the selection of the adaptation actions and
their coordination are performed by the algorithm
proposed in [
Once the set of adaptation actions is created, these actions need to be properly coordinated (when there is more than one action involved) in order to compose an adaptation strategy. This is done in Step 4, in which input and output parameters of each action are checked in order to identify immediate sequence and parallel patterns. Based on that, the attributes TotalDuration and TotalCost can be calculated. The total duration time is particularly sensible to the adopted flow pattern, sequence or parallel, for the actions execution. If the total duration time or cost exceed their constraints, the process returns to the previous step. Otherwise, the next step (Step 5) sends the created adaptation action to the adaptation parameterization module, in which the system manager shall validate the initially suggested actions parameters.
Starting from general software engineering metrics, Kaner
and Bond [
The relationships between performance indicators are identified by applying two mathematical techniques over the data matrix. The first, principal component analysis (PCA), recognizes cause-effect relations based on each indicator description. The KPIs that contain cause-effect relations are named Business Drivers Key Performance Indicators (BDKPI) because of their high factor of impact with respect the others. The second technique, partial least squares models, quantifies the importance degree of each identified cause-effect relation. These models are represented by typical regression equations that predict effect(s) from cause(s) variable(s), called PLS models, which is highly based on the designer expertise.
Taking into consideration the green aspects of the BPs,
Nowak et al. [
Service-based Applications are characterized by
independent services that, when composed, perform desired
functionalities [
Considering more comprehensive approaches, SBA
adaptation frameworks deal with large different types of service
adaptations and, in particular, propose an integrated view from
the infrastructure to the application layer [
The key point made by the authors regarding to their framework is that it takes into consideration the dependencies and effects of such actions within the three different layers. First, they tackle the lack of alignment of monitored events such that events and their mechanisms have to be related in a cross-layer manner. It enables their correlation and aggregation. Second, the lack of adaptation effectiveness is filled by providing a centralized mechanism able to aggregate and to coordinate different adaptation actions that are triggered by the same event. Third, the lack of adaptation compatibility, which means to identify adaptation necessities across layers by identifying the source of the problem that generated the event. Finally, the lack of adaptation integrity is dealt in terms of foreseeing results. It means to assess whether the selected adaptation actions are enough to achieve the desired results and how many times they need to be enacted.
Mirandola and Potena [
The solution described in this paper presents an integrated mechanism to identify and to reason about energy savings opportunities within complex environments such as data centers. It takes into consideration several aspects of a partially observable environment using knowledge-based agents. Simulated results have being performed in order to evaluate the agent behavior under stressed situations and are expected to be published in the future. A similar solution is also under evaluation which is based on a goal-based model instead of agents.
This work has been partially supported by the GAMES project (http://www.green-datacenters.eu) and Eco2Clouds EU Project (http://eco2clouds.eu), which are partially funded by the European Commission under the 7th Framework Program grant agreement numbers 248514 and 318048, respectively. This work expresses the opinions of the authors and not necessarily those of the European Commission. The European Commission is not liable for any use that may be made of the information contained in this work.