Wearable devices equipped with a camera and a display allow to develop human-centric applications providing specific services able to improve worker's productivity and increase safety in industrial environments. Despite there are diferent wearable devices in the market, current solutions are mainly focused only on passive augmented reality and real-time remote assistance. This contribution presents Artificial Intelligence technologies for wearable devices provided by NEXT VISION s.r.l. - Spin-of of the University of Catania. In particular, we present three technologies developed by NEXT VISION able to 1) localize humans in unfamiliar environments and support them to navigate the space to reach a specific destination, 2) understanding human-object interactions to provide support to the workers in industrial workplaces during complex procedures of maintenance and 3) interact with humans thanks to a conversational intelligent agent exploiting natural language and the artificial vision to answer questions regarding the surrounding environment.
mans in industrial environments improving workers’ safety and their productivity. Several vision systems sually using fixed cameras (e.g., surveillance cameras) to observe the surrounding environment from third person point of view.
More recently, diferent wearable devices equipped troduced in both consumers (e.g., Microsoft Hololens
The ability to localize workers in indoor environments
is important to support humans in performing complex
tasks and to improve safety in industrial environments
(e.g., for rescue applications). For example, a worker
could need to move from side to side in a big factory
avoiding dangerous areas with suspended loads as well
vices can provide artificial intelligence services analyzing
real world through Augmented Reality. Also these de- covered by two patents [
To support workers in these complex scenarios, NEXT Human-object interactions algorithms are fundamental
VISION developed NAIROBI (Navigating Autonomously to understand how humans relate with the surrounding
Indoor Routes by Observing Building Information), an environments while achieving their goals. In
particuArtificial Intelligence assistant able to guide the user to lar, in industrial places, the usage of specific objects and
reach specific areas (e.g., a specific workbench), or the machines need an initial training phase, often with the
closest points of interest (eg., fire extinguisher or emer- cooperation of another worker which has more
experigency exit). NAIROBI is able to localize the user in the ence, to transfer specific skills to the new workers. For
industrial environment through computational vision example, a testing procedure of a machine, may require
allowing the implementation of rescue procedures in the usage of an electric panel composed of several
butdangerous situations for the operator. The intelligent as- tons that must be pressed following a specific sequence.
sistant uses Computer Vision and Artificial Intelligence These complex operations often require people which
algorithms to localize the worker inside the building from have an expertise on the field, indeed a company has
sigimages, compute the best route and to guide the user with nificant costs in terms of time and money for providing
a path shown in the display of the wearable device using specialized training courses. Considering a complex
prothe Augmented Reality. An example of navigation pro- cedure, even an expert operator could consult a manual
vided by NAIROBI in an indoor environment is shown in during the execution of the procedure, which can slow
Figure 1. NAIROBI is also able to provide a map (i.e., “You down the task or it can lead to making mistakes.
are here” map shown through holograms on the display To support workers during the execution of complex
of the wearable device) of the indoor environment show- procedures, NEXT VISION proposes NAOMI (Next Active
ing the real-time user’s position to help users navigate a Object for Monitoring Interactions) [
NAIROBI is multiplatform and not device specific. It teractions between humans and the objects present in the surrounding environment. NAOMI is able to recognize for example to check the wear of a specific industrial the objects and the interactions with them and to guide tool in order to carry out preventive maintenance, or if the user during the execution of complex procedures sug- a specific machine has been used correctly and safely. gesting the diferent steps to perform and understanding Furthermore, the obtained information can be used to if each step has been performed correctly. Moreover, it check the quality of procedures. is able to notify alarms if an interaction is dangerous Object recognition and interaction monitoring are ad(e.g., electrick risk) avoiding mistakes which could cause dressed thanks to the artificial intelligence algorithms a machine breakdown or sending information to IoT sys- developed by Next Vision. The communication with the tems present in the environments (e.g., to immediately users through the Augmented Reality is done thanks to turn of the electric power for safety). All the informa- the Microsoft Hololens 2 capabilities (Figure 3). tion about the human-object interactions, predicted by the artificial intelligence and computer vision algorithms composing NAOMI, can be aggregated and processed to 4. Human-Machine Interaction provide statistics about the interacted objects (e.g., usage time, maintenance date, electric consumption, temperature). These information are useful to understand how the worker interacts with the surrounding environment, Artificial Intelligence allows to build systems which enable automatic conversation between humans and machines using natural language. These conversational agents could be integrated in more complex systems with the aim to support humans where they live and work. In particular, user interactions with the surrounding environment and the objects present in an industrial workplace require knowledge of specific information and procedures obtained through the training of operators with specific courses or cooperation with an expert figure.
Considering a complex environment, specific information such as “what is the testing procedure of the electric board”, “what is the next step of this maintenance procedure”, “how to calibrate the oscilloscope”, or “how to use this specific item”, may not be immediately available and require continuous interactions with laboratory technicians or qualified workers. These information are also more complex to obtain when a visual representation of the surrounding environment is needed to correctly answer to the worker’s question avoiding language ambiguity or diversity (for example “what is the object in front of me?”).
NEXT VISION developed an artificial intelligence
technologies to support operators which need to obtain
information during procedures. HERO (Human Expertise
Replication from Observation) [
The conversational assistant is able to cooperate to the other intelligent assistants developed by NEXT VISION and discussed in previous paragraphs. For example, the user can ask “guide me to the closest fire extinguisher” or “start the wizard for the object I’m looking at” to receive support from NAIROBI and NAOMI.
The proposed technologies make use of Artificial Intelligence algorithms and wearable devices equipped with a camera to improve workers productivity and to increase their safety in industrial workplaces. In particular, three artificial intelligent assistants have been presented to support workers in diferent manners. NAIROBI is able to localize the user and provide routes to reach a specific point of interest, NAOMI understands human-object interactions providing instructions on how to perform a procedure of maintenance, and HERO, a conversational assistant able to interact with workers using the natural language and artificial vision.