The applicability of Augmented Reality (AR) in business and industrial setting is gaining momentum in aspects such as ubiquitous computing, Internet of Things (IoT), and Artificial Intelligence (AI) interaction [1] [2]. The AR technology enables a person to perceive an additional layer of visual information, in which the entities are spatially and contextually correlated to real-world objects. The technological requirements include real-time object recognition and tracking in 3D space over time. Using mobile devices or smart glasses for immersive interaction with augmented information enables a myriad of new or more efficient industrial and business applications. These applications focus on presenting and interacting with just-in-time information aligned with the situational context, in order to support people's decision-making and comprehension of the environment. One of the leading applications in the industrial domain is a Remote Assistant or Advisor for field technicians, establishing a collaboration with an expert in the back-office. The remote expert is able to produce specific instructions that are sent to technician's see-through device and are anchored to the real surrounding objects for X. Franch, J. Ralyté, R. Matulevičius, C. Salinesi, and R. Wieringa (Eds.): CAiSE 2017 Forum and Doctoral Consortium Papers, pp. 33-40, 2017.
Copyright 2017 for this paper by its authors. Copying permitted for private and academic purposes. usage during motion. At present, most AR applications are hosted by the handheld tablets, but there is a strong gradient towards using smart AR glasses and helmets intended to free user's hands while enlarging the augmented field of view.
The remote advisor can be implemented in different ways [3,4,5,6], yet with similar usage and overall business goals. In particular, the aim is to:
Reduce verbal confusion with graphical and location-based instructions. Save time and increase collaboration efficiency. Reduce experts' expenses for traveling to remote sites. Enable a single remote expert to support many concurrent field technicians. Increase field operations quality. Harvest and automate capturing tacit knowledge of the aging workforce.
During the collaboration, the remote expert, located in the office, receives a visual feed from the field person. When requested, the remote expert can guide and provide textual or verbal information, including tagging augmentation points and associated information on-the-fly. This information is displayed on the device while been anchored to real world objects. In case when the remote location has not been 3D-scanned in advance, the field person needs to model the 3D scene on-the-fly or send to the remote expert some scene videos for remote 3D modeling. Consequently, the remote person can create a 3D model, annotate, create a procedure, and send it back to the field person.
Companies today employ remote assistance based on cognitive technology. Examples are Siri from Apple [7] for social interaction, Alexa from Amazon [8] for controlling home appliance or shop online, Watson Cognitive Computing from IBM [9] that enables the creation of any cognitive assistant such as for sales force, OnStar Go from General Motors [10], which is a personal assistance for automating driving actions created on IBM Watson Cognitive platform [11].
Examination of the above approaches for services driven directly by the user or by an assisting person, as well as requests from companies for services driven by cognitive Artificial Intelligence (AI), has yielded this study of an emergent repeating pattern for Remote Assistant, whether person-, self-, or cognitive-driven. This paper survey technologies, research papers, field experience and customer needs related to the pattern.
The AR supported Hybrid Remote Expert (HRE) pattern presented in this paper was identified through discussions and actual Proof-of-Concept (PoC) studies conducted with large commercial industrial partners. In addition, a survey of existing AR technology solutions currently available in the market, as well as a review of relevant academic publications, add evidence to the emergence of the pattern. A prime example of an industry looking to exploit AR for improved maintenance and repair operations is the Oil & Gas industry. It operates large, complex, capital intensive sites, where due to the hazardous nature of the materials being processed, there is a strong need for executing these tasks correctly and safely. Examples of AR based tasks include:
Procedural task guidance: the field technician carries out a maintenance or repair operation on a piece of equipment. He can use step by step instructions displayed on his AR device, observing graphical annotations on the physical equipment. An example may be "turn this valve one half turn counter-clockwise", where an arrow points to the valve and a graphic showing the turn direction is displayed. During the operation the technician may ask a question or get feedback on his actions. He might ask "What sealant fluid should I use on this flange gasket?". Again, this question might be resolved by the remote cognitive expert assistant, and if no answer can be found with a high enough level of confidence (or the system is not trained to support this task), the question will be passed on to a human remote expert. The human remote expert can see what the field technician sees (via the AR device camera) and talk naturally with field technician. The remote human expert can point to things in the field technicians' environment and the field technician will see this "remote pointing" as AR annotations attached to the physical equipment. The dialog and items pointed to during the field technician and remote human expert interaction are captured by the cognitive system and can be harvested to create new procedural guidance sequences and answered questions for future use.
Troubleshooting: the field technician is trying to determine the cause of a problem. The remote cognitive assistant will carry on a dialog with the field technician, guiding him to check the status of components using a pre-defined logic troubleshooting flowchart. At each stage, AR annotations can be used to help the field technician to understand how to check the components, similar to a step by step task guidance. If the problem is not identified using the pre-defined flowchart, then the field technician can connect with a remote human expert who can suggest further items to check until the problem is determined. As in the previous example, the suggestions by the human remote expert are captured and added to the existing knowledge base, creating troubleshooting flow chart for automated guidance the next time it is needed.
Asset Information: automatic recognition of the type and instance of a device the technician is facing (e.g. a flow meter, valve, pump) and retrieval of related information.
The above HRE usage examples are typical for all types of Process Manufacturing industrial environments (Chemical, Petroleum, Pharmaceutical, Food, Beverage, Paint, Mining…), where the right expertise is to keep the plant running smoothly.
Other industries where AR supported HRE patterns are emerging are in component manufacturing and assembly oriented industries such as Automotive, Aerospace, Electronics, Appliances, etc. In these large manufacturing plants, there are maintenance and repair operations similar to the above, that are done on the plant machinery. Yet in addition, AR can be used to aid workers in their everyday assembly jobs. For example:
1. Complex assembly and inspection guidance: AR supported HRE can be used to help guide workers through new and/or complex assembly/inspection procedures. This is especially relevant for new/reassigned workers. 2. Ad-hoc plant activities: in some cases, workers in the plant are given a task that is not part of the AR supported procedures. When help is needed, the worker can contact a remote human expert who can explain what needs to be done and clarify the instructions by annotating the workers' environment. Other areas where the HRE pattern is applicable to assist workers include:
Remote collaboration tools between two employees via video, audio, and basic annotation tools are available from technology and solutions vendors. Examples include Scope AR, which develops a remote assistance platform (the remote-AR tool) and a tool for building self-service step by step instructions for assembly and disassembly of mechanical parts (the Worklink tool) [3]. The animated instructions are built using CAD-models, intended for AR glasses. A basic collaborative method for single video streaming were proposed in [12], including annotation on a single image when instructions are needed to be presented across the sides. As an example of advanced type of information delivered by the human expert, a technology developed at XMReality enables sending mini-videos of expert's hand gestures to be visualized, illustrating for the field worker how a mission is to be conducted [4]. A work in [13] is closer to the Virtual Reality domain, offering pre-recorded videos of full body pose and motion of the expert. Other companies such as Re'flect and Inscape also use interactive two-way communication, with features including live video, anchoring and 3D tracking of expert annotations, and pre-recorded animated procedures [5,6]. A technology and interface for superimposing virtual objects, representing relevant mechanical parts, onto worker's reality are developed in works [14,15,16]. In AR context, the virtual objects are created and manipulated to build automatic AR step by step guides for complex technological processes.
Most of the applications use the display of the mobile device or AR glasses to augment information. The former occupies worker's hands and the later has issues related to workers comfort. A different approach for the visualization is taken in the IBM Research development of Tele Advisor, a versatile augmented reality tool for remote assistance [17]. In this project, a robot equipped with camera and pico-projector works side by side with the remote technician, observing remote actions and projecting annotations onto the physical environment.
Additional example of a tool enabling expert-technician interaction is the system was developed for rail industry in the ManuVAR project [18] based on the ALVAR library [19] for VR/AR applications. In addition to communication with the expert, the tool delivers data from diagnostics systems, technical details and safety information.
From the field experience and market study described above, we consider below three modes of user interaction within a Remote Assistant: self-service, person based, and cognitive based.
In self-service mode, the user of an AR-based system interacts with a fixed deterministic instructions flow on how to handle a task. The AR system identifies objects in user's field-of-view and displays a prescribed augmented item (information icon, name, small document) near the identified object. These augmented items are anchored in 3D space to the identified object. The annotations points are positioned correctly relative to the observer point of view and tracked with his motion, as seen in Figure 1. In this mode the user requests information related to anchored annotation points. Examples include: retrieving a specification sheet of a pressure valve, displaying a menu of a restaurant, performing a repair with step-by-step guided instructions (see a visualization in Figure 2), streaming live telemetry of an IoT device, or providing analytics based on measured data. The retrieved information is displayed on either smart see-through glasses, or interlaced with the mobile camera feed, mimicking a see-through user experience. Note that the AR system set up in the self-service mode does not require network connectivity since AR models and augmented information are preloaded on the device. Remote person mode is relevant when the system of 3D model, annotations and instructions is not available for the specific task, such as a custom repair action. the recommended option is to provide support with a remote human expert and augmented reality. If the object of interest has a 3D model, the remote expert can produce the annotations online, enabling the technician to visualize and follow his instructions. Additional modes of interaction are exemplified in the Remote Expert demos sited in the Prior Art section. When dealing with uncommon operations in a non-modeled environment, the 3D model can be created on-premises by online (SLAM) or offline (SfM) process, and annotated by the remote expert.
Cognitive situational context mode. The basic self-service mode with stateless query processing flow can be enhanced by interaction with an cognitive assistant system which can comprehend the situational context. Specifically, the cognitive system behavior is dependent on the history of users' interaction with the system, as well as user's interaction with recognized objects. When the context is maintained, user requests can take a form of a dialogue; when a field person asks for the temperature reading of a pipe from the remote expert, the provided answer is "90 degrees", and then the next question can be "and what is the flow direction?". In this 3 steps example, the context of the last question was taken from the previous one.
The information gathered by the system includes a task the field person is working on, former questions, gestures, and related actions, as well as historical data and IoT telemetry. The goal is to assist the field person in decision-making for understanding the environment and resolving problems. The record of user interaction path with the environment enables the system to include the data recorded from IoT objects around the path, even when they are not visible to the field person. The system is capable of processing more complex requests as it learns and accumulates information.
In a common scenario, a technician performing a repair in a machine room, needs to ensure that the power source of the engine to be repaired is disconnected. If the engine is IoT connected, the technician can point at the machine and ask the cognitive system to "power off this engine". Then the repair procedure can be guided with step by step augmented instructions. It is essential that the system kept track of user's actions in this case. Once done, the technician asks the system to reconnect the power and run a test.
The emerging pattern of the Hybrid Remote Expert combines the self-service, human expert, and cognitive assistance within a contextual analysis of the work environment. The diagram in Figure 3 depicts the flow of user interaction.
At the first step, input from user's device consisting of visual (camera) and audio (speech) information is processed by object recognition, gesture recognition and Speech-To-Text engines, determining the observed objects, user's pointing or gestures, and user`s verbal request. When the 3D model of the scene is available in advance, also the tracking of the scene and annotations anchored on physical objects are available. Analyzing user's request determines the appropriate selection of the expert mode: selfservice, human expert or a cognitive assistant. Queries for specific documents, specification sheets, and pre-recorded guidance, are available through the self-service route. More complex queries requiring context analysis, real-time operational information driven from telemetry or analytics sources, and more, are routed to the cognitive expert service. Finally, if an automatic solution is not available, human expert is invoked.
A possible realization of the cognitive expert mode is based on services of IBM Watson Cognitive platform. When activated, the cognitive system analyzes the situation using the categories and logic provided by the human experts, based on observed objects, current requests and history of conversation. The provided cognitive response ranges from answering questions regarding the current state and IoT telemetry, through retrieval of expert knowledge pertinent to the situation, to step-by-step procedures for standard maintenance and troubleshooting operations.
The human back-office expert intervenes when necessary, assisting the technician by a live video chat, producing on-the-fly annotations and hand drawings for the observed scene, and anchoring annotations to physical locations. The activity is logged by in order to maintain the situation context and to be used in future interaction. Fig. 3. Flow chart describing system components and the flow of user interaction with the hybrid remote expert.
Business value of Augmented Reality is evident in terms of saved time of field workers, reduced cost of experts, global reach for market penetration, as well as reduction in penalty or business loss due to slow operations by non-experienced workers. These values are repetitive across industrial domains, and the needs are resolved similarly. This similar approach for providing a conceptual solution with different implementations either driven by people, self-service, or cognitive assistant, is recognized as the emerging pattern of a Hybrid Remote Expert (HRE). Practical solutions implementing HRE should be developed on top of a "build your own AR" cloud middleware, so that business units and services can create their own valuable solutions, using the variety of technologies such as hand/pointing/gesture recognition, and a powerful text analysis engine. These technologies enable higher-level tasks like guiding and monitoring user actions, thus assisting the user in more complex scenarios.