-For Industry 4.0 - the Internet of Things (IoT) in an industrial manner - new methodologies for support and collaboration of employees are needed. One of these methodologies combines existing work practices with support through technologies like Augmented Reality (AR). Therefore, usability concepts for appropriate hardware as well as the data transfer need to be analyzed and designed within applicable industry standards. In this paper, we present two different use cases (Real-Time Machine Data Overlay and Web-Based AR Remote Support) in the context of collaboration and support of employees. Both use cases are focusing on three main requirements: 1) Effective data transmission; 2) Devices certified for industrial environments; and 3) Usability targeted towards industrial users. Additionally, we present an architecture recommendation for combining both use cases as well as a discussion of the benefits and the limitations of our approaches leading to future directions.
Many current Augmented Reality use cases target
commercial and industrial areas. These have the biggest market
potential of the enterprise segment according to the forecast
by Goldman Sachs [
Typical scenarios involve remote assistance and monitoring. These use cases yield a very measurable return on investment, thereby easing the creation of a business case to offset the research and development costs. Sending specialized engineers to remote locations to fix issues with broken machines is more expensive than developing and providing means for efficient remote collaboration using an on-site generalist, interacting with the remote specialist.
However, the main challenge is designing and creating a suitable software architecture and user interface. Thus, we explicitly focus on the following in this paper: 1) Efficient data transmission for mobile scenarios, involving collaboration and / or real-time machine data display. 2) Software should run on durable and cheap devices that are certified for industrial environments . Most current AR devices are targeted for home or entertainment use. 3) Usability targeted towards industrial users, to reduce required training time to a minimum.
To provide guidance for these challenges, we have developed two prototypes: “Web-Based AR Remote Support” and “Real Time Machine Data Overlay”.
AR applications for industry got serious research attention
since the early 1990’s. [
The advantages of mobile AR for visual analytics for
industrial IoT (IIoT) data in a networked environment have been
widely recognized [
Remote support and collaboration in industrial settings
have been discussed for example in [
Based on the requirements of two Industry 4.0 companies, we have developed two separate prototypes to address their real-life issues. In this section, we first summarize the individual learnings regarding the challenges highlighted in the introduction. Then, we abstract the architecture to provide our recommendations for a generic framework, following bestpractices learned through the individual projects.
The first prototype allows placing holographic dashboards in the real world to visualize real-time machine data retrieved through industry standard protocols (Real-Time Machine Data Overlay). The second enables a remote expert to draw support annotations on the AR camera view of an on-site generalist (Web-Based AR Remote Support).
In Fig. 1, a user placed an information dashboard on a pipe.
Data is retrieved from an OPC Unified Architecture (OPC
UA) server. The HoloLens app can show a list of available
nodes, whose dashboards can be placed on real-world
structures. Interaction is performed with the standardized HoloLens
select gesture using two fingers. The user interface is mainly
composed of Holographic Buttons from the MR Toolkit [
The overlay of the most important information directly on the machines in a manufacturing hall allows identifying potential issues with a quick glimpse. The main challenge is combining usability with efficient data transfer (see Fig. 2).
The architecture is based on three separate parts: 1) Company IT containing data storage and processing servers; 2) Mixed Reality (MR) headset handling user interaction and data visualization (VIS); and 3) Actual physical world anchoring of VIS objects.
1) Data Transmission / Company IT: Common interoperability standards for machine data monitoring are OPC UA and Message Queuing Telemetry Transport (MQTT).
In the scenario of an untethered MR headset in a factory, data transmission quality and speed can be an issue. As such, a relay server has two advantages: it acts as an additional security layer, as there is no need to expose the OPC UA endpoints to mobile clients in the wireless network. Moreover, it only sends the necessary data to the mobile client to reduce data traffic.
D u¨rkop et. al [
2) Devices / AR Headset: The most critical part of the application runs within the actual headset. On the technical side, it needs to register with the server and update its internal database of accessible OPC Node IDs. These are shown in menus, allowing the user to freely place dashboards in the real world. Each dashboard mainly needs to store the connected Node ID and the current dataset in addition to a world anchor.
To be used within industry environments, careful checks
need to be performed if the headset fulfills necessary safety
standards. The HoloLens already complies to several
standards: ANSI Z87.1, CSA Z94.3 and EN 166 [
3) Usability / Physical World & Persistence: A key
consideration of AR in industrial scenarios is the structure of
the real-world environment. The underlying computer vision
algorithms from Google ARCore and Microsoft HoloLens
use a Simultaneous Localization and Mapping (SLAM) based
approach to create a geometrical reconstruction of the world
while at the same time estimating camera localization. Even
though the commercial algorithms are not available, a
stateof-the-art open source implementation is ORB-SLAM [
Based on this limitation, AR applications should not
encourage placing dashboards on feature-less walls (e.g.,
singlecolored with little structure). A better approach is directly
placing the items on machines, typically having a more
complex structure. In addition, early research from Boeing [
The world anchor is managed through the Microsoft MR
toolkit and essentially forms the connection of the virtual
objects to the physical world. Nevertheless, a persistence
manager component within the app needs to ensure the persistence
across sessions, and potentially also across multiple users
simultaneously viewing the same scene with different headsets.
To avoid instability, virtual objects are recommended to be
placed at a maximum distance of 3 m from an anchor [
Special attention needs to be paid to the legibility of the
dashboards. Research shows that diegetic and spatial user
Remote Support
interfaces are the most natural and preferred metaphors for
virtual scenarios [
To put usability first, billboard functionality is used for user
oriented dashboard visualization. The downside is that parts
of the dashboard might appear to be “inside” the physical
object. However, due to the semi-transparent material and
the deactivated occlusion with the spatial map, that effect
is hardly noticeable in our prototype. To optimize anchoring
the virtual dashboard on the real-world object, the prototype
additionally allows re-positioning through hand-based drag
gestures. Manual label placement allows contextual relevance
for users and avoids challenges of automated placement [
The purpose of remote support via AR is that two persons – a customer who needs help on a certain technical problem and a remote expert (supporter) who provides a solution on that issue – can exchange visual information (see Fig. 3).
The initial starting point is a video stream of the customer’s environment. The customer shows the point of interest (POI, describes the object where the actual problem is located) to the supporter. In industrial environments this could be a malfunctioning machine. AR technology allows the supporter to give the customer interactive visual feedback (e.g., by making annotations), which is anchored to the POI and stays in place.
For our system we identified two core requirements during the conceptual phase: first, the supporter should have no need to install additional software. Second, no special hardware should be required – neither from the customer nor from the supporter. We decided to use mobile devices (smartphones or tablets) for the customer, as these are widespread and available in industry-certified variants that can be used in environments like production halls.
Overall, the system should be universally applicable, as well as easily available for a huge potential target group.
1) Devices & Installation: In our scenario, the customer is willing to install new software on his mobile device, if it helps him to get quick and efficient support. However, the effort for that installation has to be low. After installation, the app should be usable without the need for configuration.
The supporter was defined as an expert with special knowledge on certain machines or technical devices. He maybe travels a lot (e.g., doing installations). There’s a chance that he has no possibility or time to install software when the customer calls for quick help.
To satisfy these requirements, we designed a system where the supporter gets an individual hyperlink (via E-Mail) from the customer. After opening the link, the supporter receives the real-time video stream from the customer within his web browser. There, he can add visual feedback, environmentally linked to the Augmented Reality POI.
2) Data Transmission: The underlying technology for the data transmission of the stream and the graphical annotations is WebRTC. WebRTC is a web standard for building peer-topeer connections between two browsers or between a browser and another application that supports an implementation of WebRTC. During the stream, there is no need for an additional node or logical overhead in-between (e.g., a streaming server).
To establish the connection, the customer’s app automatically registers with a signaling server and receives a session ID. The app then generates the individual hyperlink, which includes the new session ID. The customer sends that hyperlink to the supporter (e.g., via E-Mail). By opening the hyperlink, the supporter’s web browser also connects to the signaling server. Thus, the two peers exchange their Interactive Connectivity Establishment (ICE) information via the signaling server and establish a direct peer-to-peer connection. Next, the customer streams the environment to the supporter by using the rear camera of his mobile device. At the same time, the device performs environmental understanding on the captured video input using ARCore, attempting to find key-points.
3) Usability & Collaboration: For drawing on a video stream, there are two different concepts: (a) The drawings can be integrated in real-time into the currently active stream. This could cause inaccuracies when the customer moves the device while the supporter is drawing. (b) Pause the video stream in the supporter’s browser during drawing.
Based on these two general approaches, the focus of the project on accurate annotations and usage within industry environments leads to approach (b): whenever the supporter performs a tap on the streamed video, the last frame freezes in the browser. The supporter then has the possibility to draw annotations on the frozen frame. At the same time, the customer can continue to move his phone without influencing the supporter’s view.
At the same time, a transparent plane is created as AR element in the environment of the customer. By setting an anchor (using the detected environmental key-points), the plane stays at the designated location near the POI. The annotation will later appear on that plane.
Remote Support
AR Assistance Client
On completion, the supporter confirms his annotations. hand, smartphone-based AR works on industry-ready phones, The annotation data is directly sent to the customer. There, which are easier to carry and cheaper to roll out to employees it is used to create an image texture, which is added to in service and production. Overall, implementing a common the previously generated plane. The supporter’s annotation shared persistence back-end with specialized clients for the becomes visible on the plane and is anchored to the POI. The different scenarios gives the best of both worlds. annotations appear in the customer’s environment.
IV. RECOMMENDATIONS FOR AR APP ARCHITECTURE
These two individual use cases are tightly related to each other. Combined, they lead to a complete use case, while still retaining the unique architectural challenges of each part.
While our approach A describes the initial dashboard view for live machine data, approach B then allows connecting to a remote expert / supporter in case issues become evident.
In Fig. 4 we show an architecture unifying both use cases.
It exhibits two main differences: 1) The client’s responsibility is focused only on interaction handling – allowing the users to add new dashboards, or to draw annotations; 2) The persistence is centralized in the server. This creates a shared database, allowing improved multi-user support, as well as a more seamless transition between both use cases. For example, improvements suggested by the supporter could be immediately seen in the real-time machine data dashboards.
This centralized architecture is based on the most recent developments to share anchors between users and platforms.
While SLAM-algorithms from researchers have already been
optimized for collaborative SLAM [
Based on the initial observations from our prototypes, we see the distinction between different specialized clients as an important factor. Head-mounted displays generally have a higher cost and might be less comfortable to wear for a whole working day, but offer better immersion and precision (e.g., through time-of-flight depth sensing). Such a device would be suitable for example for the shift manager to get a quick overview of dashboards of various machines. On the other
Time Machine Data Overlay and for Web-Based AR Remote Support), meeting the future challenges of Industry 4.0. Based on the two approaches and considering our lessons learned, we proposed a future design approach for a combined architecture (see Sec. IV).
Since our prototypes have only been evaluated at a very low
level (non-documented discussions with domain experts), it is
necessary to perform a design study [
The insights generated through our two prototypes,
combined with previously gathered experience of collaborative and
multi-device scenarios [