We present HoloMiracle, a system that enables operators of industrial equipment and beyond to pose queries about physical, virtual, regulatory, and functional relationships between components of the equipment and that visualizes the responses to their queries in-situ, as a holographic overlay. We report on HoloMiracle's system architecture and discuss a concrete use case in the automotive manufacturing domain.
Devices and systems across industrial domains { from manufacturing lines and turbines to transformers and buildings { have virtual and physical components that interact with each other in a plethora of di erent ways: physically, components might be connected to one another structurally which might imply that phenomena such as vibrations are transmitted across components of a device; virtually, di erent components might be associated to a common process { consequently, recon gurations of one station in a manufacturing line might a ect others; there might also be regulatory connections between components, e.g., between all zones in a building that must be covered by smoke detectors, and components might be functionally similar, meaning that one might be used in place of another. Currently, operators access such information about industrial equipment through various software applications, typically on a desktop or laptop computer, or in the form of physical documentation. The information retrieval process from these sources can take a considerable amount of time and is prone to errors, as their information back ends are often not integrated. device. This enables the system to render responses to complex queries directly on top of a physical device or system and makes the wealth of information that is stored in complex, potentially cross-domain knowledge models, accessible to average users in an intuitive and e cient way.
HoloMiracle was deployed in the context of interacting with a model of an automotive manufacturing line that mirrors a line deployed by Miracle Automation Engineering Co., Ltd., a leading manufacturer of logistics automation technology and equipment. We use this speci c example to illustrate the functionality of our system, although the approach described is in principle applicable to any complex system of any granularity. In this demonstration, HoloMiracle produces a visual rendering of the components of the manufacturing line from a semantic model that includes the line's components (e.g., skid rail, skid, motors, wheels, sensors, etc.), their spatial dimensions, behavioral and functional properties, and connectivity. An operator uses a cursor and a speech interface to interact with the assembly line and pose queries that are answered by the underlying semantic model and can be as complex as the model is detailed. Examples of such queries are "Show all proximity sensors" { the system highlights all components of type ProximitySensor (yellow), see Fig. 1(a) { and "Show all directly connected components" { the system highlights all components (yellow) that are directly connected to a component selected using the cursor (blue), see Fig. 1(b). The system can also answer deeper queries such as "Show all sensors that can measure vibration of the selected component" by highlighting those acceleration sensors that are suitable for measuring the vibration of a speci c component.
We expect that HoloMiracle should contribute to making the interaction with and exploration of a system more natural and e cient for humans, as it enables the in-situ querying of equipment and rendering of information for "onthe-spot" decisions. In addition to shortening the time spent on asset surveys, assessment, and analytics, and enabling site engineers to gain more insight in the target system, it should also prove valuable for recording service processes to identify potential for future improvements.