=Paper=
{{Paper
|id=Vol-1848/CAiSE2017_Forum_Paper5
|storemode=property
|title=Hybrid Remote Expert - an Emerging Pattern of Industrial Remote Support
|pdfUrl=https://ceur-ws.org/Vol-1848/CAiSE2017_Forum_Paper5.pdf
|volume=Vol-1848
|authors=Ethan Hadar,Joseph Shtok,Benjamin Cohen,Yochay Tzur,Leonid Karlinsky
|dblpUrl=https://dblp.org/rec/conf/caise/HadarSCTK17
}}
==Hybrid Remote Expert - an Emerging Pattern of Industrial Remote Support==
https://ceur-ws.org/Vol-1848/CAiSE2017_Forum_Paper5.pdf
Hybrid remote expert - an emerging pattern of industrial
remote support
Ethan Hadar1,2, Joseph Shtok2, Benjamin Cohen2, Yochay Tzur2,
Leonid Karlinsky2
1 Zafed Academic College, Zafed, Israel, ethan@zefat.ac.il
2 IBM Research Labs, Haifa, Israel
{ethan, josephs, cohen, yochayt, leonidka}@il.ibm.com
Abstract. One of today’s challenges in the Industrial domain is to support work-
ers decision-making and comprehension of the situation by enhancing and ex-
tending workers vision. This paper examines a pattern of industrial needs and
related solutions for technical support and information-based guidance. The in-
formation is provided to field technicians in order to repair and maintain equip-
ment on site, using Augmented Reality (AR) applications running on smart
glasses or tablets. The pattern suggests three main interconnected modes of con-
suming contextual technical information with Augmented Reality: (1) assisted by
pre-recorded augmented information, (2) guided by a remote human expert, or
(3) supported by an autonomous cognitive system. The pattern, emerged by har-
vesting our field experience and a literature review, is termed as a hybrid remote
expert; in the paper, we describe how it is observed in the industry and discuss
the business uses.
Keywords: Augmented Reality, Context Awareness, Remote Support, Context
and customer needs.
1 Introduction
The applicability of Augmented Reality (AR) in business and industrial setting is
gaining momentum in aspects such as ubiquitous computing, Internet of Things (IoT),
and Artificial Intelligence (AI) interaction [1][2]. The AR technology enables a person
to perceive an additional layer of visual information, in which the entities are spatially
and contextually correlated to real-world objects. The technological requirements in-
clude real-time object recognition and tracking in 3D space over time. Using mobile
devices or smart glasses for immersive interaction with augmented information enables
a myriad of new or more efficient industrial and business applications. These applica-
tions focus on presenting and interacting with just-in-time information aligned with the
situational context, in order to support people’s decision-making and comprehension of
the environment. One of the leading applications in the industrial domain is a Remote
Assistant or Advisor for field technicians, establishing a collaboration with an expert in
the back-office. The remote expert is able to produce specific instructions that are sent
to technician’s see-through device and are anchored to the real surrounding objects for
X. Franch, J. Ralyté, R. Matulevičius, C. Salinesi, and R. Wieringa (Eds.):
CAiSE 2017 Forum and Doctoral Consortium Papers, pp. 33-40, 2017.
Copyright 2017 for this paper by its authors. Copying permitted for private and academic purposes.
usage during motion. At present, most AR applications are hosted by the handheld tab-
lets, but there is a strong gradient towards using smart AR glasses and helmets intended
to free user’s hands while enlarging the augmented field of view.
The remote advisor can be implemented in different ways [3,4,5,6], yet with similar
usage and overall business goals. In particular, the aim is to:
Reduce verbal confusion with graphical and location-based instructions.
Save time and increase collaboration efficiency.
Reduce experts’ expenses for traveling to remote sites.
Enable a single remote expert to support many concurrent field technicians.
Increase field operations quality.
Harvest and automate capturing tacit knowledge of the aging workforce.
During the collaboration, the remote expert, located in the office, receives a visual
feed from the field person. When requested, the remote expert can guide and provide
textual or verbal information, including tagging augmentation points and associated in-
formation on-the-fly. This information is displayed on the device while been anchored
to real world objects. In case when the remote location has not been 3D-scanned in
advance, the field person needs to model the 3D scene on-the-fly or send to the remote
expert some scene videos for remote 3D modeling. Consequently, the remote person
can create a 3D model, annotate, create a procedure, and send it back to the field person.
Companies today employ remote assistance based on cognitive technology. Exam-
ples are Siri from Apple [7] for social interaction, Alexa from Amazon [8] for control-
ling home appliance or shop online, Watson Cognitive Computing from IBM [9] that
enables the creation of any cognitive assistant such as for sales force, OnStar Go from
General Motors [10], which is a personal assistance for automating driving actions cre-
ated on IBM Watson Cognitive platform [11].
Examination of the above approaches for services driven directly by the user or by
an assisting person, as well as requests from companies for services driven by cognitive
Artificial Intelligence (AI), has yielded this study of an emergent repeating pattern for
Remote Assistant, whether person-, self-, or cognitive-driven. This paper survey tech-
nologies, research papers, field experience and customer needs related to the pattern.
2 Background
The AR supported Hybrid Remote Expert (HRE) pattern presented in this paper was
identified through discussions and actual Proof-of-Concept (PoC) studies conducted
with large commercial industrial partners. In addition, a survey of existing AR technol-
ogy solutions currently available in the market, as well as a review of relevant academic
publications, add evidence to the emergence of the pattern. A prime example of an in-
dustry looking to exploit AR for improved maintenance and repair operations is the Oil
& Gas industry. It operates large, complex, capital intensive sites, where due to the
hazardous nature of the materials being processed, there is a strong need for executing
these tasks correctly and safely. Examples of AR based tasks include:
34
Procedural task guidance: the field technician carries out a maintenance or repair
operation on a piece of equipment. He can use step by step instructions displayed on his
AR device, observing graphical annotations on the physical equipment. An example
may be “turn this valve one half turn counter-clockwise”, where an arrow points to the
valve and a graphic showing the turn direction is displayed. During the operation the
technician may ask a question or get feedback on his actions. He might ask “What seal-
ant fluid should I use on this flange gasket?”. Again, this question might be resolved by
the remote cognitive expert assistant, and if no answer can be found with a high enough
level of confidence (or the system is not trained to support this task), the question will
be passed on to a human remote expert. The human remote expert can see what the field
technician sees (via the AR device camera) and talk naturally with field technician. The
remote human expert can point to things in the field technicians’ environment and the
field technician will see this “remote pointing” as AR annotations attached to the phys-
ical equipment. The dialog and items pointed to during the field technician and remote
human expert interaction are captured by the cognitive system and can be harvested to
create new procedural guidance sequences and answered questions for future use.
Troubleshooting: the field technician is trying to determine the cause of a prob-
lem. The remote cognitive assistant will carry on a dialog with the field technician,
guiding him to check the status of components using a pre-defined logic troubleshooting
flowchart. At each stage, AR annotations can be used to help the field technician to
understand how to check the components, similar to a step by step task guidance. If the
problem is not identified using the pre-defined flowchart, then the field technician can
connect with a remote human expert who can suggest further items to check until the
problem is determined. As in the previous example, the suggestions by the human re-
mote expert are captured and added to the existing knowledge base, creating trouble-
shooting flow chart for automated guidance the next time it is needed.
Asset Information: automatic recognition of the type and instance of a device the
technician is facing (e.g. a flow meter, valve, pump) and retrieval of related information.
The above HRE usage examples are typical for all types of Process Manufacturing
industrial environments (Chemical, Petroleum, Pharmaceutical, Food, Beverage, Paint,
Mining…), where the right expertise is to keep the plant running smoothly.
Other industries where AR supported HRE patterns are emerging are in component
manufacturing and assembly oriented industries such as Automotive, Aerospace, Elec-
tronics, Appliances, etc. In these large manufacturing plants, there are maintenance and
repair operations similar to the above, that are done on the plant machinery. Yet in ad-
dition, AR can be used to aid workers in their everyday assembly jobs. For example:
1. Complex assembly and inspection guidance: AR supported HRE can be used to help
guide workers through new and/or complex assembly/inspection procedures. This is
especially relevant for new/reassigned workers.
2. Ad-hoc plant activities: in some cases, workers in the plant are given a task that is
not part of the AR supported procedures. When help is needed, the worker can con-
tact a remote human expert who can explain what needs to be done and clarify the
instructions by annotating the workers’ environment.
Other areas where the HRE pattern is applicable to assist workers include:
35
Transportation maintenance & repair: aircraft, trains, cars, ships,…
Energy & utilities: power plants, utility grids, transformer stations,…
Construction: matching plans to what needs to be built, inspection, …
3 Related Work
Remote collaboration tools between two employees via video, audio, and basic an-
notation tools are available from technology and solutions vendors. Examples include
Scope AR, which develops a remote assistance platform (the remote-AR tool) and a
tool for building self-service step by step instructions for assembly and disassembly of
mechanical parts (the Worklink tool) [3]. The animated instructions are built using
CAD-models, intended for AR glasses. A basic collaborative method for single video
streaming were proposed in [12], including annotation on a single image when instruc-
tions are needed to be presented across the sides. As an example of advanced type of
information delivered by the human expert, a technology developed at XMReality en-
ables sending mini-videos of expert’s hand gestures to be visualized, illustrating for the
field worker how a mission is to be conducted [4]. A work in [13] is closer to the Virtual
Reality domain, offering pre-recorded videos of full body pose and motion of the ex-
pert. Other companies such as Re’flect and Inscape also use interactive two-way com-
munication, with features including live video, anchoring and 3D tracking of expert
annotations, and pre-recorded animated procedures [5,6]. A technology and interface
for superimposing virtual objects, representing relevant mechanical parts, onto
worker’s reality are developed in works [14, 15, 16]. In AR context, the virtual objects
are created and manipulated to build automatic AR step by step guides for complex
technological processes.
Most of the applications use the display of the mobile device or AR glasses to aug-
ment information. The former occupies worker’s hands and the later has issues related
to workers comfort. A different approach for the visualization is taken in the IBM Re-
search development of Tele Advisor, a versatile augmented reality tool for remote as-
sistance [17]. In this project, a robot equipped with camera and pico-projector works
side by side with the remote technician, observing remote actions and projecting anno-
tations onto the physical environment.
Additional example of a tool enabling expert-technician interaction is the system
was developed for rail industry in the ManuVAR project [18] based on the ALVAR
library [19] for VR/AR applications. In addition to communication with the expert, the
tool delivers data from diagnostics systems, technical details and safety information.
4 Forms of AR based interaction
From the field experience and market study described above, we consider below
three modes of user interaction within a Remote Assistant: self-service, person based,
and cognitive based.
In self-service mode, the user of an AR-based system interacts with a fixed deter-
ministic instructions flow on how to handle a task. The AR system identifies objects in
36
user’s field-of-view and displays a prescribed augmented item (information icon, name,
small document) near the identified object. These augmented items are anchored in 3D
space to the identified object. The annotations points are positioned correctly relative to
the observer point of view and tracked with his motion, as seen in Figure 1. In this mode
the user requests information related to anchored annotation points. Examples include:
retrieving a specification sheet of a pressure valve, displaying a menu of a restaurant,
performing a repair with step-by-step guided instructions (see a visualization in Figure
2), streaming live telemetry of an IoT device, or providing analytics based on measured
data. The retrieved information is displayed on either smart see-through glasses, or in-
terlaced with the mobile camera feed, mimicking a see-through user experience. Note
that the AR system set up in the self-service mode does not require network connectivity
since AR models and augmented information are preloaded on the device.
Fig. 1. Visualization of the scene-rooted AR annotations in technical environment, given as ar-
rows and yellow dots with textual labels.
Remote person mode is relevant when the system of 3D model, annotations and
instructions is not available for the specific task, such as a custom repair action. the
recommended option is to provide support with a remote human expert and augmented
reality. If the object of interest has a 3D model, the remote expert can produce the an-
notations online, enabling the technician to visualize and follow his instructions. Addi-
tional modes of interaction are exemplified in the Remote Expert demos sited in the
Prior Art section. When dealing with uncommon operations in a non-modeled environ-
ment, the 3D model can be created on-premises by online (SLAM) or offline (SfM)
process, and annotated by the remote expert.
Cognitive situational context mode. The basic self-service mode with stateless
query processing flow can be enhanced by interaction with an cognitive assistant sys-
tem which can comprehend the situational context. Specifically, the cognitive system
behavior is dependent on the history of users’ interaction with the system, as well as
user’s interaction with recognized objects. When the context is maintained, user
Fig. 2. Snapshots from a guided step-by-step procedure for detecting and repairing an elec-
tronic board fault.
37
requests can take a form of a dialogue; when a field person asks for the temperature
reading of a pipe from the remote expert, the provided answer is “90 degrees”, and
then the next question can be “and what is the flow direction?”. In this 3 steps exam-
ple, the context of the last question was taken from the previous one.
The information gathered by the system includes a task the field person is working
on, former questions, gestures, and related actions, as well as historical data and IoT
telemetry. The goal is to assist the field person in decision-making for understanding
the environment and resolving problems. The record of user interaction path with the
environment enables the system to include the data recorded from IoT objects around
the path, even when they are not visible to the field person. The system is capable of
processing more complex requests as it learns and accumulates information.
In a common scenario, a technician performing a repair in a machine room, needs to
ensure that the power source of the engine to be repaired is disconnected. If the engine
is IoT connected, the technician can point at the machine and ask the cognitive system
to “power off this engine”. Then the repair procedure can be guided with step by step
augmented instructions. It is essential that the system kept track of user’s actions in this
case. Once done, the technician asks the system to reconnect the power and run a test.
5 Hybrid Remote Expert - emerging pattern
The emerging pattern of the Hybrid Remote Expert combines the self-service, human
expert, and cognitive assistance within a contextual analysis of the work environment.
The diagram in Figure 3 depicts the flow of user interaction.
At the first step, input from user’s device consisting of visual (camera) and audio
(speech) information is processed by object recognition, gesture recognition and
Speech-To-Text engines, determining the observed objects, user’s pointing or gestures,
and user`s verbal request. When the 3D model of the scene is available in advance, also
the tracking of the scene and annotations anchored on physical objects are available.
Analyzing user’s request determines the appropriate selection of the expert mode: self-
service, human expert or a cognitive assistant. Queries for specific documents, specifi-
cation sheets, and pre-recorded guidance, are available through the self-service route.
More complex queries requiring context analysis, real-time operational information
driven from telemetry or analytics sources, and more, are routed to the cognitive expert
service. Finally, if an automatic solution is not available, human expert is invoked.
A possible realization of the cognitive expert mode is based on services of IBM Wat-
son Cognitive platform. When activated, the cognitive system analyzes the situation
using the categories and logic provided by the human experts, based on observed ob-
jects, current requests and history of conversation. The provided cognitive response
ranges from answering questions regarding the current state and IoT telemetry, through
retrieval of expert knowledge pertinent to the situation, to step-by-step procedures for
standard maintenance and troubleshooting operations.
38
The human back-office expert intervenes when necessary, assisting the technician
by a live video chat, producing on-the-fly annotations and hand drawings for the ob-
served scene, and anchoring annotations to physical locations. The activity is logged
by in order to maintain the situation context and to be used in future interaction.
Fig. 3. Flow chart describing system components and the flow of user interaction with the hybrid
remote expert.
6 Afterword
Business value of Augmented Reality is evident in terms of saved time of field work-
ers, reduced cost of experts, global reach for market penetration, as well as reduction
in penalty or business loss due to slow operations by non-experienced workers. These
values are repetitive across industrial domains, and the needs are resolved similarly.
This similar approach for providing a conceptual solution with different implementa-
tions either driven by people, self-service, or cognitive assistant, is recognized as the
emerging pattern of a Hybrid Remote Expert (HRE). Practical solutions implementing
HRE should be developed on top of a “build your own AR” cloud middleware, so that
business units and services can create their own valuable solutions, using the variety of
technologies such as hand/pointing/gesture recognition, and a powerful text analysis
engine. These technologies enable higher-level tasks like guiding and monitoring user
actions, thus assisting the user in more complex scenarios.
39
References
1. Cook, T. On Augmented Reality. Businessinsider.com: http://www.businessinsider.com/ap-
ple-ceo-tim-cook-explains-augmented-reality-2016-10, referenced on Dec 4 2016
2. Heath N. Five ways augmented reality will transform your business. TechRepublic.
http://www.zdnet.com/article/five-ways-augmented-reality-will-transform-your-business/
(2016)
3. Scope AR company, RemoteAR and Worklink collaboration tools:
http://www.scopear.com/products, referenced on Dec 4 2016.
4. XMReality company, Remote guidance demo.
https://www.youtube.com/watch?v=tmp5KNCn0RE referenced on Dec 4 2016.
5. Re`flect company, remote assistance demo. https://www.re-flekt.com/industrial/, referenced
on Dec 4 2016.
6. Inscape company: remote assistance tool demo
https://www.youtube.com/watch?v=M1mKffgNK98, referenced on Dec 4 2016.
7. Apple company, Siri Assistant: http://www.apple.com/ios/8/, referenced on Dec 4, 2016.
8. Amazon company, Alexa assistant: http://9.amazon.com/spa/index.html, referenced on Dec
4 2016.
9. IBM company, Watson Cognitive Assistant: http://www-01.ibm.com/common/ssi/cgi-
bin/ssialias?htmlfid=GBJ03056USEN, referenced on Dec 4 2016.
10. General Motors, OnStar Go platform: https://techcrunch.com/2016/10/26/gm-puts-ibm-
watson-in-cars-with-the-new-11-go-platform/, referenced on Dec 4 2016.
11. IBM Company, IBM Watson providing platform for the OnStar: https://www-
03.ibm.com/press/us/en/pressrelease/50838.wss, referenced on Dec 4 2016.
12. Kuzuoka, H.: Spatial workspace collaboration: a SharedView video support system for re-
mote collaboration capability. In Proceedings of the SIGCHI conference on Human factors
in computing systems, ACM, pp. 533-540 (June 1992)
13. Kurillo, G., Bajcsy, R., Nahrsted, K. and Kreylos, O.:. Immersive 3d environment for remote
collaboration and training of physical activities. In 2008 IEEE Virtual Reality Conference,
pp. 269-270. (March 2008)
14. Michalos, G., Karagiannis, P., Makris, S., Tokçalar, Ö. and Chryssolouris, G.: Augmented
Reality (AR) Applications for Supporting Human-robot Interactive Cooperation. Procedia
CIRP, 41, pp.370-375. (2016)
15. Oda, O., Elvezio, C., Sukan, M., Feiner, S. and Tversky, B.: Virtual Replicas for Remote
Assistance in Virtual and Augmented Reality. In Proceedings of the 28th Annual ACM
Symposium on User Interface Software & Technology. pp. 405-415. ACM (Nov 2015)
16. Henderson, S.J. and Feiner, S.K. Augmented reality for maintenance and repair (ARMAR).
Report. COLUMBIA UNIV NEW YORK DEPT OF COMPUTER SCIENCE. (2007)
17. Gurevich, P., Lanir, J., Cohen, B. and Stone, R.: TeleAdvisor: a versatile augmented reality
tool for remote assistance. In Proceedings of the SIGCHI Conference on Human Factors in
Computing Systems (pp. 619-622). ACM (May 2012)
18. Azpiazu J., Siltanen S., Multanen P., Mäkiranta A., Barrena N., Diez A., Agirre J. and Smith
T.: Remote support for maintenance tasks by the use of Augmented Reality: the 24 project,
Congress on Virtual Reality Applications CARVI (2011).
19. ALVAR, A Library for Virtual and Augmented Reality, http://www.vtt.fi/multimedia/al-
var.html, referenced on Dec 4, 2016.
40