=Paper=
{{Paper
|id=Vol-3043/short8
|storemode=property
|title=Industry 4.0 Resource Monitoring - Experiences With Micrometer and Asset Administration Shells
|pdfUrl=https://ceur-ws.org/Vol-3043/short8.pdf
|volume=Vol-3043
|authors=Miguel Gomez Casado,Holger Eichelberger
|dblpUrl=https://dblp.org/rec/conf/kpdays/CasadoE21
}}
==Industry 4.0 Resource Monitoring - Experiences With Micrometer and Asset Administration Shells==
https://ceur-ws.org/Vol-3043/short8.pdf
Industry 4.0 Resource Monitoring - Experiences With
Micrometer and Asset Administration Shells
Miguel G. Casadoa,b , Holger Eichelbergerb
a
University of Valladolid, 47002 Valladolid, Spain
b
Software Systems Engineering, Universität Hildesheim, 31141 Hildesheim, Germany
Abstract
Industry 4.0/IIoT setups are often distributed and consist of hundreds of sensors, devices or machines.
Nowadays, powerful edge devices enable the execution of software services like Artificial Intelligence
close to production machines to reduce latency. Although several supporting Industry 4.0 platforms do
exist, industrial stakeholders aim for more open, interoperable and vendor-neutral solutions.
In this paper, we focus on the monitoring of runtime properties in Industry 4.0 for services and (edge)
devices. We propose a realization based on existing components like Micrometer, MQTT and upcoming
standards such as the Asset Administration Shell. We discuss patterns for their integration, the impact
of the patterns on performance in terms of an experiment as well as experiences that we made so far.
Keywords
Industry 4.0, IIoT, monitoring, Micrometer, performance, Asset Administration Shell, BaSyx, MQTT
1. Introduction
The digitization of industry increases the performance of technical systems and their processes,
but also their complexity. Particular complexity arises as Industry 4.0 installations are inher-
ently distributed. In such settings, edge devices allow for handling the high data frequencies
and volumes provided by sensors and production machines. Moreover, modern edge devices
enable the safe execution of hard real-time machine control operations and (soft real-time) IT
functionality, such as Artificial Intelligence (AI).
In the BMWi-funded project IIP-Ecosphere1 , we research approaches to address typical
shortcomings in Industry 4.0 systems regarding openness, interoperability, or vendor-neutrality.
At the heart of IIP-Ecosphere, we develop a prototypical AI-enabling Industry 4.0 platform
that allows for exploring the approaches developed in the project. As a core functionality,
the platform shall automatically deploy and manage (AI-)services in a uniform manner on
heterogeneous resources such as edge devices, on-premise servers, or cloud resources.
Within this context, we focus in this paper on a runtime monitoring approach for (AI-)services.
Our research question is: How to design and realize efficient resource and service monitoring in
an open, interoperable and vendor-neutral manner? Our approach combines a recent Industry
SSP’21: Symposium on Software Performance, November 09–10, 2021, Leipzig, Germany
Envelope-Open miguel.gomez@alumnos.uva.es (M. G. Casado); eichelberger@sse.uni-hildesheim.de (H. Eichelberger)
GLOBE http://www.sse.uni-hildesheim.de (H. Eichelberger)
Orcid 0000-0002-2584-5558 (H. Eichelberger)
© 2021 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
Workshop
Proceedings CEUR Workshop Proceedings (CEUR-WS.org)
http://ceur-ws.org
ISSN 1613-0073
1
https://www.iip-ecosphere.eu/
4.0 information/interface model, the Asset Administration Shell (AAS) [1], with protocols such
as MQTT or AMQP2 and a vendor neutral monitoring frontend (Micrometer3 ).
For these components, we introduce three integration patterns and analyze them in a perfor-
mance experiment. While a local integration of monitoring and AAS performs best, it implies
also higher resource usage, a potential obstacle for resource-constrained edge devices. In con-
trast, a distributed installation increases response times. We show that a combination with
publish-subscribe protocols helps balancing resource usage and response time.
Structure of the paper: In Section 2, we provide background information on used standards
and technologies. We present our approach to Industry 4.0 resource and service monitoring in
Section 3 and discuss initial experimental results in Section 4. In Section 5, we briefly outline
related work, and in Section 6 we conclude the paper and give an outlook on future work.
2. Background
In this section, we introduce standards and technologies that we selected to meet the goals of
IIP-Ecosphere. For the realization, we aim at maximizing the use of Open Source.
Micrometer is an Open Source vendor-neutral application metrics facade with support for
monitoring tools like Prometheus or Dynatrace. Micrometer defines so-called meters such as
timers, gauges, or counters, which also carry their unit of monitoring, e.g., items/s.
A range of protocols is used in IIoT, including (legacy) machine protocols such as Fieldbus (the
field level is not in our scope as stated in [2]) to Message Queuing Telemetry Transport (MQTT)
or Advanced Message Queuing Protocol (AMQP). MQTT and AMQP are publish-subscribe
protocols for the transport of binary payloads via a broker server.
ŶĞƐƚŝŶŐůĞǀĞůƐ >ĞŐĞŶĚ
ƉůĂƚĨŽƌŵ EĞƐƚŝŶŐ
ƌĞƐŽƵƌĐĞƐ ^ƚĂƚŝĐƉƌŽƉĞƌƚLJ
ĂϬϬϱϬϱϲϬϬϬϬϴ LJŶĂŵŝĐƉƌŽƉĞƌƚLJǁͬŽŐĞƚƚĞƌ͕ƐĞƚƚĞƌ
ƐLJƐƚĞŵͺŵĞŵŽƌLJͺƵƐĂŐĞсϬ͘ϰϴ ƐƵďͲŵŽĚĞů KƉĞƌĂƚŝŽŶĐĂƵƐŝŶŐƌĞŵŽƚĞĐĂůů
ƐLJƐƚĞŵͺĐƉƵͺƵƐĂŐĞсϬ͘ϭϱ ĞůĞŵĞŶƚ ƐƵďͲŵŽĚĞů ZĞĨĞƌĞŶĐĞ
ƐƚĂƌƚ^ĞƌǀŝĐĞ;ŝĚ͗^ƚƌŝŶŐͿ ĐŽůůĞĐƚŝŽŶ ^
͙
ƐĞƌǀŝĐĞƐ
ĂϬϬϱϬϱϲϬϬϬϬϴͺĨĂŝůƵƌĞWƌĞĚŝĐƚŽƌ
ƌĞƐŽƵƌĐĞсĂϬϬϱϬϱϲϬϬϬϬϴ
ƚŚƌŽƵŐŚƉƵƚсϴϵϮ
͙
Figure 1: Structure of an Asset Administration Shell, here for compute resources and services.
Recently, the Asset Administration Shell (AAS) [1] was defined to uniformly model In-
dustry 4.0 ”assets”, i.e., products, machines, their digital twins, or even software components.
AAS is currently in a standardization driven by industrial groups in different countries, e.g., the
Industrial Digital Twin Association (IDTA) in Germany or the Smart Manufacturing Profiles
initiative in the U.S. Although AAS is rather new and still in development, the IIP-Ecosphere
2
https://mqtt.org/, https://www.amqp.org/
3
https://micrometer.io/
partners opted for an extensive exploration. Essentially, as illustrated in Figure 1, an AAS is a
nested structure containing typed properties, operations or references between elements. An
AAS can be static, dynamic regarding values or structures, or active through callable operations.
Eclipse BaSyx4 is the de-facto AAS reference implementation. In BaSyx, an AAS is represented
as a REST structure that can be served by a local process or deployed to a remote server.
Properties and operations can be realized through functors attached to an AAS that perform
local or remote method invocation, e.g., via the BaSyx Virtual Automation Bus (VAB) protocol.
3. Approach
Our goal is to collect runtime monitoring data reflecting the resource usage and performance
of devices and services in Industry 4.0 installations. A realization shall provide access to the
monitored information for individual devices and services, but also serve as a basis for efficient
aggregation over all elements in an installation. Moreover, IIP-Ecosphere imposes further
requirements [2], among them in particular: 1) AAS shall be used for all component/service
interfaces, 2) IIoT protocols such as MQTT or AMQP can be used instead of AAS, e.g., for soft
realtime streaming, 3) monitoring operations shall be faster than a typical 8 ms machine pace,
4) (de facto) standards shall be used wherever possible to foster interoperability.
To realize the monitoring approach, we made some basic decisions (cf. [3] for more details):
For representing runtime measures in an uniform fashion, we rely on Micrometer. However, Mi-
crometer focuses on local meters, i.e., there is limited built-in support to access remote/published
meters. Thus, we propose a set of proxy instances, which can be loaded with JSON information
obtained from published meters, e.g., through a REST client. Further, the monitoring results
shall be represented in terms of AAS structures as illustrated in Figure 1, which may also
contain JSON. This allows client implementations, e.g., the AASX Package Explorer5 , to access
monitored properties through polling as desired by our stakeholders. Alternative forms, e.g.,
pushing of the data through publish-subscribe, can be realized side-by-side.
/ŶƋƵŝƌĞƌ͕Ğ͘Ő͕͘ĂŐŐƌĞŐĂƚŝŽŶ͕^yWĂĐŬĂŐĞ džƉůŽƌĞƌ ĞǀŝĐĞƐ͕Ğ͘Ő͕͘ĞĚŐĞ͕ƐĞƌǀĞƌ͕ĐůŽƵĚ >ĞŐĞŶĚ͗
ůŽĐĂůĐĂůů
;ĂͿ Z ^d DŝĐƌŽŵĞƚĞƌ ^ĞƌǀŝĐĞ ƌĞŵŽƚĞĐĂůů
ůŝĞŶƚ DŝĐƌŽŵĞƚĞƌ WƌŽdžŝĞƐ ^ ^ĞƌǀŝĐĞ ƉƵďͲƐƵď
;ďͿ ůŝĞŶƚ DŝĐƌŽŵĞƚĞƌ WƌŽdžŝĞƐ Z ^d ^ s s/ŵƉů͘^ĞƌǀĞƌ DŝĐƌŽŵĞƚĞƌ ^ĞƌǀŝĐĞ
^ĞƌǀŝĐĞ
Z ^d ƉƵďͲƐƵď DŝĐƌŽŵĞƚĞƌ
;ĐͿ ůŝĞŶƚ DŝĐƌŽŵĞƚĞƌ WƌŽdžŝĞƐ ^ ĂĐŚĞ ^ĞƌǀŝĐĞ
^ĞƌǀŝĐĞ
ĐŚĂŶŶĞů Đ
Figure 2: AAS integration patterns for AAS-based runtime monitoring.
The mentioned components can be combined in several ways, implying individual perfor-
mance impacts. Figure 2 illustrates some alternative patterns, all including services with defined
4
https://www.eclipse.org/basyx/
5
https://www.eclass.eu/anwendung/verwaltungsschale.html
meters on the device side, either a local or remote AAS to represent the monitored data, and a
client reading the remote meters via Micrometer proxies. Our patterns are:
a) A local AAS executing the Micrometer environment, which, in turn, observes the services.
The AAS polls the monitored data when a property is accessed, i.e., its functors perform
local calls. To publish the AAS, an HTTP server process must be running on the device.
b) A remote AAS communicating through a (light-weight) AAS implementation server on
the device. As in a), the data is polled, here through (VAB) remote method calls.
c) Remote pub-sub AAS: A scheduled task on the device publishes the data on a channel
𝑐. The AAS subscribes to 𝑐 and holds the received data in a local cache bound to the
AAS property functors. If communication from the AAS to the device is needed, e.g., to
reconfigure the monitoring, a protocol as in b) can additionally be used.
As a BaSyx AAS is represented as a REST structure, all patterns access the AAS from the client
through REST. Patterns a) and b) exclusively rely on AAS functionality and particularly follow
the examples provided by BaSyx. In contrast, in pattern c) we connect device and AAS/client
using asynchronous communication and transform the polling from pattern b) into an approach
that relies on local calls to a cache instead of remote method invocations. In this pattern, further
components can subscribe to the channel and operate even without access to the AAS. Patterns
a) and b) may also be combined with some form of pushing, which we do not detail here.
4. Results
To analyze the efficiency of the patterns introduced in Section 3, we present now an experimental
evaluation of the individual response times. We realized the patterns as variants of the same
Java code6 . As basis, we use a simple workload in terms of two interconnected Spring Cloud
Stream services (Spring version 3.1.1). The Micrometer (version 1.6.4) instances represent more
than 50 default Spring and custom meters. A BaSyx AAS (github version of January 2021)
defines a representative structure of properties and operations for all variants. A client reads
the AAS through our Micrometer proxies and records the individual response times.
We use the local AAS pattern as baseline. However, Spring and BaSyx conflict when being
executed in the same JVM. Thus, we use corresponding meters without service workload as
fallback baseline. For the remote AAS pattern, the client polls meter data via VAB. As the meters
are published by Spring in terms of REST by default, we opted for a VAB implementation server
that accesses the meter information through an internal REST client [3]. For the pub-sub AAS,
a regular task publishes a JSON summary of the meters via AMQP (HiveMQ client 2020.4).
As experimental environment, we use Windows 10 (Dell 7490, Intel i7-8650U, 1.9MHz, 32
GByte RAM, Open JDK 13+33) to emulate a development setting and Ubuntu Linux 20.04
(VMWare VM with 6 vCores on a Xeon E5-2650v4, 2.2GHz, 12 GByte RAM, Open JDK 13.0.7)
as a server environment. Both setups use local network only. A single iteration of a variant
reads 62 AAS properties representing gauges, counters and timers. Initial experiments in [3]
6
Replication package https://doi.org/10.5281/zenodo.5571817
indicated, that the measurements stabilize at around 175 iterations, probably due to caching
or JVM/JIT settling operations. Here, we perform 200 iterations for each variant. To prevent
warm-up outliers, we exclude the results of first two iterations per variant.
12
Windows Linux
Gauge response time [ms}
avg [ms] avg [ms] 10
local a) 1,3 0,3 0,4 0,1
Gauges VAB b) 8
5,5 1,1 2,2 0,7
AMQP c) 1,2 0,6 0,7 0,3 6
local a) 0,8 0,3 0,4 0,2
Timer VAB b) 5,5 1,2 2,6 1,0 4
AMQP c) 1,6 0,8 1,0 0,8 2
local a) 0,6 0,3 0,3 0,2
Counter VAB b) 3,6 0,9 1,7 0,8 0
local VAB AMQP local VAB AMQP
AMQP c) 1,1 0,5 0,6 0,3
Windows Windows Windows Linux Linux Linux
Figure 3: Evaluation results, response times for all patterns (left) and box plots for gauges (right).
Figure 3 summarizes the results, in particular response times of the three variants for both,
Windows and Linux, as well as representative boxplots (min/max-whiskers) for the gauges,
currently the most relevant meter for us. As the VAB-based implementation of pattern b)
involves three protocols (AAS-REST, VAB, Spring-REST), it is not surprising that the response
times are factor 4-7 higher than the baseline. Additional experiments show that this difference
can roughly be attributed in equal amounts to VAB and Spring-REST. If we replace those two
protocols by publish-subscribe in pattern c), we can achieve significantly better response times
(factor 1-2.5 of the baseline) and less fluctuations, while still not serving the AAS on the device.
Further, we found that gauges may incur a response time drop of factor 2-3, if they are updated
on request rather than asynchronously, e.g., by a schedule. Moreover, each AAS access in BaSyx
creates a temporary network connection. A high access frequency may cause an outage of
system resources requiring a cool-down of 120 seconds between two experiment runs.
The measures depend on the used components and the setup, which may limit generalizability.
Although the use of REST or VAB may be rather specific here, we believe that the remote pub-sub
pattern can be beneficial in similar (REST) settings. Comparable results can also be expected for
MQTT, as other work in IIP-Ecosphere indicates that MQTT and AMQP behave rather similar
for the load applied here. Moreover, the Windows setup is less representative, as it was mainly
intended to support the judgement of ad hoc measures during development.
5. Related Work
We addressed a combination Industry 4.0 protocols, Micrometer and AAS. For IIoT protocols,
a body of work is published with a certain focus on MQTT. Two interesting related works
are [4] on broker performance for IoT edge computing and [5] on MQTT protocol latency over
different gateways. Regarding Micrometer, publications typically focus on applications, e.g., in
the context of Spring [6]. Further, there is an increasing amount of publications on AAS and,
in particular, BaSyx. Currently, the focus there is on evaluating maintainability [7] or on use
cases/demonstrators [8]. According to our knowledge, there is currently no work that combines
AAS, Micrometer and IIoT protocols for the monitoring of resource or software services.
6. Conclusions
Industry 4.0 systems do not only require complex, distributed setups, but sometimes also the use
of prescribed protocols and standards. Moreover, future Industry 4.0 systems aim at increased
openness, interoperability and vendor-neutrality. In this context, we analyzed whether Asset
Administration Shells, Micrometer and protocols such as MQTT or AMQP can be combined to
realize efficient runtime monitoring of resource and service properties. We proposed three basic
integration patterns and discussed the impact on the performance in terms of an experiment.
In essence, a more complex publish-subscribe implementation can outperform techniques that
are (mostly) shipped with the utilized implementation frameworks. Moreover, efficient updates
of meters may require specific attention, e.g., asynchronous schedules.
In future, we plan to explore the scalability of the approach when monitoring a larger set of
devices. On the one side, an overview of all monitored values through an Asset Administration
Shell is demanded by our stakeholders and can be provided as shown in this work. On the other
side, we expect that a realization of further platform components, e.g., a monitoring aggregator,
can efficiently be achieved through an integrated publish-subscribe approach.
Acknowledgments
IIP-Ecosphere is partially supported by the German Ministry of Economics and Energy (BMWi),
grant 01MK20006D.
References
[1] S. Bader, E. Barnstedt, H. Bedenbender, et al., Details of the Asset Administration Shell, 2020.
URL: https://www.plattform-i40.de/PI40/Redaktion/DE/Downloads/Publikation/Details_
of_the_Asset_Administration_Shell_Part1_V3.html.
[2] H. Eichelberger, C. Sauer, A. S. Ahmadian, M. Schicktanz, A. Dewes, G. Palmer, C. Niederée,
IIP-Ecosphere Platform Requirements, 2021. doi:1 0 . 5 2 8 1 / z e n o d o . 4 4 8 5 7 7 4 .
[3] M. G. Casado, Service and device monitoring on devices in IIP-Ecosphere, 2021. URL:
https://sse.uni-hildesheim.de/studium-lehre/beispiele-guter-ausarbeitungen/.
[4] H. Koziolek, S. Grüner, J. Rückert, A Comparison of MQTT Brokers for Distributed IoT
Edge Computing, in: Software Architecture, 2020, pp. 352–368.
[5] S. B. Kenitar, S. Marouane, A. Mounir, A. Younes, A. G. Gonzalez, Evaluation of the MQTT
Protocol Latency over Different Gateways, in: Intl. Conf. on Smart City Applications, 2018.
[6] F. Gutierrez, Monitoring, Apress, 2021, pp. 383–397.
[7] K. Esper, F. Schnicke, Evaluation of the Maintainability Aspect of Industry 4.0 Service-
oriented Production, in: Intl. Conf. on Industry 4.0, Artificial Intelligence, and Communica-
tions Technology, 2020, pp. 8–14.
[8] F. Schnicke, T. Kuhn, P. O. Antonino, Enabling Industry 4.0 Service-Oriented Architecture
Through Digital Twins, in: Software Architecture, 2020, pp. 490–503.