To cope with the increased complexity, physical systems are more and more supported by software systems consisting of multiple subsystems. Usually, each of the subsystems uses standards relevant to the subsystem for interoperability with other tools. Thus, one faces the problem that information about the system as a whole is distributed across multiple models. To solve this problem, model views can be introduced to combine these models and extract application-specific knowledge. As an example, the smart grid is a cyber-physical system where one is interested to detect, manage and prevent system outages. The information necessary to do this is split among the standards IEC 61970/61968, IEC 61850 and IEC 62056. This paper presents a benchmark case and evaluation framework for joining information spread across multiple models into a single view, based on a model-based outage management system for smart grids. Because cyber-physical systems often require very fast response times to changes of underlying models, the benchmark focuses especially on the incremental computation of model views.
The complexity of todays systems, for example cyber-physical systems, makes it inevitable to divide the system into multiple subsystems that operate in different domains. In many of these domains, standards exist that the respective subsystem has to comply with or for which many tools can be reused.
For example, the smart grid is a cyber-physical system that spans the physical structures of the electricity
network and the system of software systems that monitor, control, and repair the system in case of outages [
Because each of these standards describe different aspects of the system, models according to these standards have to be combined if multiple aspects are required to gain some insights about the system. Applying modeldriven engineering, model views are a tool to extract information from multiple models without confronting the user with unnecessary information for a particular purpose, for example analysis.
In the area of smart grids, an additional challenge is the size of the models and the frequency of changes. In combination, this means that very large amounts of data have to be processed in a very short amount of time. However, the changes usually only affect small parts of the model, which is why an incremental view computation appears beneficial.
In this paper, we propose a benchmark based on selected model views created by Mittelbach and Burger [
In the area of smart grids, the relevant standards are IEC 61970/61968, IEC 61850 and IEC 62056. These
standards are briefly described below (taken from [
IEC 62056 COSEM (Companion Specification for Energy Metering) is the international standard for data
exchange for meter reading, tariff and load control in the domain of electricity metering. It works together
with the Device Language Message Specification (DLMS). Together, they provide a communication
profile to transport data from metering equipment to the metering system and to define a data model and
communication protocols for data exchange [
While these standards are useful in their domain, one has to combine the information represented by these
standards to detect and prevent outage situations. Mittelbach and Burger presented a model-based outage
system that synchronizes models of these standards and consists of a set of 15 views to help operators to manage
outage situations [
In the scope of the proposed benchmark, we focus on two model views contained in the model-based outage management system. A rather simple view is created to detect outages, while a second slightly more complex view supports the prevention of outages.
For both tasks, we present the original implementation of the view in ModelJoin [
To detect an outage, we use the fact that a smart meter cannot send any data when it is cut off power supply. If this happens, the system can try to reach the meter but will receive a connection failure notification. This is used to detect outages without relying on customer feedback.
The information that a connection to a smart meter is lost is depicted in the CIM model. The relevant excerpt for this task is depicted in Figure 1b. It has to be matched with the corresponding physical devices in the COSEM model where its location is stored. The latter is depicted in Figure 1c.
PositionPoint xPosition : EString yPosition : EString zPosition : EString [0..1] Position
Location [0..1] Location
EnergyConsumer Reachability : EInt PowerA : EDouble = 0.0 ID : EString
UUID : EString
EnergyConsumer [0..1] EnergyConsumer ServiceDeliveryPoint [0..*] ServiceDeliveryPoints
IdentifiedObject mRID : EString
[0..*] Assets [0..1] AssetContainer
[0..1] Location Asset [0..*] Assets (a) The Viewtype for the Outage De- [0..1] ServiceDeliveryPoint
[0..*] EndDeviceAssets tection Task
(b) Excerpt from the CIM metamodel relevant for task 1
COSEMRoot [0. *] PhysicalDevice
PhysicalDevice ID : EString
[0. 1] AutoConnect [0. 1] ElectricityValues
AutoConnectObject Connection : EBoolean = false
ElectricityValues
ApparentPowermL1 : EDouble = 0.0 (c) Excerpt from the COSEM metamodel relevant for task 1
A correct solution should match the meter assets in the CIM model with the physical devices in the COSEM model. For each of these matches, the result model should contain a model element that conforms to the generated view type class EnergyConsumer. For this element, a range of (partially derived) attributes shall be copied and the reference to the location of meter asset and physical device should be saved.
This reference to location and position point shall respect referential integrity. This means, if two meter assets in the CIM model reference the same location, their joins in the view should also reference the same Location element in the view.
Element
UUID : EString
The analysis algorithms to detect system disturbances proposed in [
Task 2 requires to match elements from all three domain standards. For the COSEM standard, the used metamodel excerpt is very similar to Task 1. The relevant metamodel excerpts for CIM and the substation standard are depicted in Figure 2a and Figure 2b, respectively.
The analysis viewtypes will not provide the analysis result but only the matrix of phasor data for the
comparison. Six queries were defined in [
mRID␣=␣substationStandard.LNNodes.LNGroupM.MMXU.NamePlt.IdNs" as jointarget.PMUVoltageMeter { keep attributes CIM.IEC61970.Core.IdentifiedObject.mRID keep calculated attribute "substationStandard.LNNodes.LNGroupM.MMXU.PhV.phsA.cVal.mag.f" as PMUVoltageMeter.VoltageAMag:Double keep calculated attribute "substationStandard.LNNodes.LNGroupM.MMXU.PhV.phsA.cVal.ang.f" as PMUVoltageMeter.VoltageAAng:Double keep calculated attribute "substationStandard.LNNodes.LNGroupM.MMXU.PhV.phsB.cVal.mag.f" as PMUVoltageMeter.VoltageBMag:Double keep calculated attribute "substationStandard.LNNodes.LNGroupM.MMXU.PhV.phsB.cVal.ang.f" as PMUVoltageMeter.VoltageBAng:Double keep calculated attribute "substationStandard.LNNodes.LNGroupM.MMXU.PhV.phsC.cVal.mag.f" as PMUVoltageMeter.VoltageCMag:Double keep calculated attribute "substationStandard.LNNodes.LNGroupM.MMXU.PhV.phsC.cVal.ang.f" as PMUVoltageMeter.VoltageCAng:Double keep calculated attribute "substationStandard.LNNodes.LNGroupM.MMXU.PhV.neut.cVal.mag.f" as PMUVoltageMeter.VoltageNeutMag:
Double keep calculated attribute "substationStandard.LNNodes.LNGroupM.MMXU.PhV.neut.cVal.ang.f" as PMUVoltageMeter.VoltageNeutAng:
Double keep calculated attribute "substationStandard.LNNodes.LNGroupM.MMXU.PhV.net.cVal.mag.f" as PMUVoltageMeter.VoltageNetMag:Double keep calculated attribute "substationStandard.LNNodes.LNGroupM.MMXU.PhV.net.cVal.ang.f" as PMUVoltageMeter.VoltageNetAng:Double keep calculated attribute "substationStandard.LNNodes.LNGroupM.MMXU.PhV.res.cVal.mag.f" as PMUVoltageMeter.VoltageResMag:Double keep calculated attribute "substationStandard.LNNodes.LNGroupM.MMXU.PhV.res.cVal.ang.f" as PMUVoltageMeter.VoltageResAng:Double keep supertype CIM.IEC61968.Assets.Asset as type jointarget.Asset { keep outgoing CIM.IEC61968.Assets.Asset.Location as type jointarget.Location { keep outgoing CIM.IEC61968.Common.Location.Position as type jointarget.PositionPoint { keep attributes CIM.IEC61968.Common.PositionPoint.xPosition, CIM.IEC61968.Common.PositionPoint.yPosition,
CIM.IEC61968.Common.PositionPoint.zPosition } keep outgoing CIM.IEC61968.Common.Location.PowerSystemResources as type jointarget.PowerSystemResource { keep subtype CIM.IEC61970.Core.ConductingEquipment as type jointarget.ConductingEquipment { keep outgoing CIM.IEC61970.Core.ConductingEquipment.Terminals as type jointarget.Terminal { keep outgoing CIM.IEC61970.Core.Terminal.TieFlow as type jointarget.TieFlow {
keep outgoing CIM.IEC61970.ControlArea.TieFlow.ControlArea as type jointarget.ControlArea { subtypes of some model elements. If an energy consumer in a service delivery point is a ConformLoad, then the view computation should be different to the case when the energy consumer is a NonConformLoad. 4
The benchmark framework is based on the benchmark framework of the TTC 2015 Train Benchmark case [
1. Initialization: In this phase, solutions may load metamodels and other infrastructure independent of the used models as required. Because time measurements are very hard to measure for this phase, the time measurement is optional.
LoadGroup [0.1] SubLoadArea
SubLoadArea NonConformLoadGroup
ConformLoadGroup [0.1] LoadGroup NonConformLoad [0.1] LoadGroup
[0.1] ControlArea ConformLoad
ControlArea mRID : EString [0.1] LoadArea
LoadArea [0.1] ControlArea 4. Updates: A sequence of change sequences is applied to the model. Each change sequence consists of several atomic change operations. After each change sequence, the view must be consistent with the changed source models, either by entirely creating the view from scratch or by propagating changes to the view result.
In the following subsections, the change sequences, solution requirements and the benchmark configuration are explained in more detail. 4.1
To measure the incremental performance of solutions, the benchmark uses generated change sequences. These change sequences are in the changes directory of the benchmark resources. Additional change sequences can be generated using a generator contained in the benchmark resources, however the generator is implemented using
The changes are available in the form of models. An excerpt of the metamodel is depicted in Figure 4: There are classes for each elementary change operation that distinguish between simple assignments and collection interactions, such as adding or removing single elements or resetting, which means erasing the collection contents. The true metamodel contains concrete classes that distinguish further between the type of feature, whether it is an attribute, association or composition change. In these concrete classes, the added, deleted or assigned items are included1. The change metamodel also supports change transactions where a source change implies some other changes, for example setting opposite references.
Unfortunately, the implementation to apply these changes is only available in NMF. Incremental tools are therefore asked to transform the change sequences into their own change representation. To ease the specification of batch solutions, the generator also outputs the models after each change sequence step. 1In a composite insertion, the added element is contained, otherwise only referenced.
ModelChangeSet invertible : EBoolean = false
ElementaryChange affectedElement : EObject
ElementaryChangeTransaction
ListInsertion index : EInt feature : EStructuralFeature
CollectionInsertion feature : EStructuralFeature
CollectionReset feature : EStructuralFeature
CollectionDeletion feature : EStructuralFeature
PropertyChange feature : EStructuralFeature
ListDeletion index : EInt feature : EStructuralFeature The solutions are required to perform the steps of the benchmark and report the following metrics after each step, in case of the update phase after every change sequence.
Solutions should report on the runtime of the respective phase in integer nanoseconds (Time), the working set in bytes (Memory) and the root element count in the created view (Elements). The memory measurement is optional. If it is done, it should report on the used memory after the given phase (or iteration of the update phase) is completed. Solutions are allowed to perform a garbage collection before memory measurement that does not have to be taken into account into the times. In the update phase, we are not interested in the time to load models or changes or the perhaps required transformation of changes, but only the pure view update, i.e. either recomputation of the view or propagation of the change.
To enable automatic execution by the benchmark framework, solutions should add a subdirectory to the solutions folder of the benchmark with a solution.ini file stating how the solution should be built and how it should be run. An example configuration for the ModelJoin solution is depicted in Listing 3. Because the solution contains the already compiled Jar archive, no action is required for build. However, solutions may want to run build tools like maven in this case to ensure the benchmark runs with the latest version. 1 [build] 2 default=echo ModelJoin solution is already compiled 3 skipTests=echo The ModelJoin solution is already compiled 4 5 [run] 6 OutageDetection=java -Xmx8G -jar solution.jar -view OutageDetection 7 OutagePrevention=java -Xmx8G -jar solution.jar -view OutagePrevention
Listing 3: An example solution.ini file 1 { 2 3 4 5 6 7 8 9 10 11 12 13 } "Views": [ "OutageDetection", "OutageAvoidance" ], "Tools": ["ModelJoin"], "ChangeSets": [
"changeSequence1" ], "Sequences": 100 "SequenceLength": 10 "Runs": 5 5. 1 &> python scripts/run.py 5
The repetition of executions as defined in the benchmark configuration is done by the benchmark. This means, for 5 runs, the specified command-line for a particular view will be called 5 times. These runs should all have the same prerequisites. In particular, solutions must not save intermediate data between different runs. Meanwhile, all iterations of the Update phase are executed in the same process and solutions are allowed (and encouraged) to save any intermediate computation results they like, as long as the results are correct after each change sequence.
The root path of the input models and changes, the run index, the number of iterations and the name of change sequences are passed using environment variables ChangePath, RunIndex, Sequences and ChangeSet. To demonstrate the usage of these environment variables, the benchmark framework also contains a demo solution which does nothing but print out a time csv entry using the provided environment variables. The benchmark framework only requires Python 2.7 or above and R to be installed. R is required to create diagrams for the benchmark results. Furthermore, the solutions may imply additional frameworks. We would ask solution authors to explicitly note dependencies to additional frameworks necessary to run their solutions.
If all prerequisits are fulfilled, the benchmark can be run using Python with the command python scripts/run.py. Additional command-line options can be queried using the option –help.
The benchmark framework can be configured using JSON configuration files. The default configuration is depicted in Listing 4. In this configuration, both views are computed, using only the reference solution in ModelJoin, running the change sequence changeSequence1 contained in the changes directory 5 times each. The exact commands created by the benchmark framework are determined using the solution configuration files described below.
To execute the chosen configuration, the benchmark can be run using the command line depicted in Listing
Additional commandline parameters are available to only update the measurements, create visualizations or generate new change sequences.
Solutions of the proposed benchmark should be evaluated by their completeness, correctness, conciseness, understandability, batch performance and incremental performance.
For each evaluation, a solution can earn 5 points for Task 1 and 5 points for Task 2. In the latter, we explain how the points are awarded for Task 1. The points for Task 2 are awarded equivalently. 5.1
Assessing the completeness and correctness of model transformations is a difficult task. In the scope of this benchmark, besides manual assessment by opponents, solutions are checked for the correct number of model elements in the result after each change.
Detecting and especially forecasting an outage in a smart grid heavily relies on heuristics. Therefore, it is important to specify views in a concise manner.
0 points The task is not solved.
To evaluate the conciseness, we ask every solution to note on the lines of code of their solution. This shall include the model views and glue code to actually run the benchmark. Code to convert the change sequence can be excluded. For any graphical part of the specification, we ask to count the lines of code in a HUTN2 notation of the underlying model. 5.3
Similarly to conciseness, it is important for maintainance tasks that the solution is understandable. However, as there is no appropriate metric for understandability available, the assessment of the understandability is done manually. For solutions participating in the contest, this score is collected using questionnaires at the workshop. For the performance, we consider two scenarios: batch performance and incremental performance. For the batch performance, we measure the time the solution requires to create the view for existing models. For the incremental solution, we measure the time for the solution to propagate a given set of changes. Points are awarded according to the following rules: 0 points The task is not solved.
The measurements for batch performance and for incremental performance are done separately. This means, solutions are allowed to run in a different configuration when competing for the batch performance than they use in the incremental setting. This is because in an application, usually only one of these aspects is particularly important. 5.5
Due to their importance, the points awarded in completeness & correctness and understandability are doubled in the overall evaluation. Furthermore, due to the importance of incremental updates, we give the incremental performance a double weight.
2http://www.omg.org/spec/HUTN/ We would like to thank Victoria Mittelbach for the permission to use her master thesis results for this case.