New synergies between human and Cyber Physical Systems (CPS) are key aspects for generating collaborators instead of competitors, strengthening the human role of Industry 5.0. Bidirectional Communication Channels (BCCs) between humans and machines lay the foundation of these synergies within manufacturing environments. This position paper reviews main currently available visualization and interaction technologies to connect data, humans, CPS, Artificial Intelligence (AI) and machines, presenting a selection of use cases and applications for AI services. The successful design, development and integration of these bidirectional communication channels poses relevant open research challenges for the future to revolutionize current manufacturing environments.
Industry 4.0 is a paradigm, centered around the emergence of cyber-physical objects, and offering
a promise of enhanced efficiency through digital connectivity and Artificial Intelligence (AI).
According to the European Commission, Industry 4.0, as currently conceived, “is not fit for purpose
in a context of climate crisis and planetary emergency, nor does it address deep social tensions” [
Although nowadays the Industry 4.0 vision is not a reality, the European Union envisions the
Industry 5.0 as a new paradigm evolving a production and consumption driven economic model into a
more transformative view of growth that is focused on human progress and well-being based on
reducing and shifting consumption to new forms of sustainable, circular and regenerative economic
value creation and equitable prosperity. The objective is to “seek people-planet-prosperity, combining
competitiveness and sustainability, rather than simply value extraction to benefit shareholders” [
One of the main objectives of Industry 5.0 is to bring back human workers to the factory floors,
generating synergies combining the human brainpower and creativity with the automation and AI
technologies of semi and/or fully autonomous machines [
A Bidirectional Communication Channel (BCC) is a requirement to generate these synergies,
leading to effective human-in-the-loop systems. Human-in-the-loop is an anthropocentric mechanism,
which allows a direct sharing or transfer of human skills to a subset of CPS control loops [
To effectively collaborate with humans, CPS must adequately understand human intention and
desire. Moreover, humans must have the means to understand, analyze and trust predictions and
actions from CPS. According to [
For example, [
Focusing on HMI, [
Visualization and interaction technologies are the foundation of these BCCs for this new generation HMIs, connecting humans with machines, CPS, data and AI services. Although these technologies are already available to use within Industry 4.0, their effective integration is still an open research topic. This position paper reviews main currently available technologies to generate these BCCs, presenting some use cases and applications to encourage further research in this area.
This section introduces the most promising visualization and interaction technologies for the generation of BCCs for Industry 5.0. Although the integration of these technologies into new generation interfaces is an open research area, examples of the individual use of the technologies already exist. Figure 1 summarizes the contribution and the approach followed by this paper.
At the left corner, different human profiles interacting with industrial systems are represented. In
production systems, two main reference models for human activities have been identified:
Human-inthe-Loop and Human-in-the-Mesh. The first one, traditionally related to operators, refers to situations
in which the worker is directly participating in the process of products fabrication or assembling and
its loop of control. The second one, traditionally related to managers and engineers, refers to
situations where the worker participates in production planning and it loop of control [
This BCC is composed by both Human-to-Machine and Machine-to-Human channels. Although not all the visualization and interaction technologies are suitable for both channels, their integration will foster the development of new BCCs empowering workers through the use of digital devices and endorsing a human centric approach to technology.
Visual Computing has already been identified as a key enabling technology for Industry 4.0.
Visual Analytics, Augmented and Virtual Reality, Computer Vision, HCI and related technologies are
central to many disruptive applications in a Smart Factory perspective [
These technologies connect humans with the main elements of Industry 5.0 (Figure 1). The CPS
represents the core communication element, as it merges the physical and the virtual world,
connecting both to machines and to their digital representation, commonly named as digital twin. CPS
capture data from the elements and feed AI services executed either at the edge, cloud or at both.
Industry 5.0 aligns the objective of these AI services towards a sustainable, human-centric and
resilient industry [
This paper focuses on the following visualization and interaction technologies: visual analytics; augmented, virtual and mixed reality; voice recognition; natural language processing and gesture recognition.
Visual Analytics (VA) combines the strength of human cognitive abilities with analysis methods
to extract information from data. High-dimensional, real-time visualization allows the graphical
expression of complex process variables at a fraction of the cost of full-scale digitalization [
Challenges faced by VA include (i) the integration of process relevant analytics and visualization,
and (ii) the integration of aging workforce. Aligned with Industry 5.0 objectives, measures should be
taken to ensure the ease of use and increased accessibility of VA, with minimal training and
upskilling required to gain access to intuitive data visualization [
Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR) technologies are
shaping new interaction environments. These technologies, which integrate physical and virtual
objects at different levels, have received several definitions [
Voice recognition and natural language processing systems have greatly increased their performance within the last years due to the integration of new models based on deep neural networks. Conversational assistants, such as general-purpose Apple Siri, Google Now, Microsoft Cortana or Amazon Alexa, have become the main example of this improvement, simplifying and making human-machine interaction more natural. Voice assistants could act as a central interaction technology for Industry 5.0. As they are natural for workers, they require minimal training, lowering the knowledge barrier for existing workforce to interact with CPS. Moreover, they are both eyes-free and hands-free, allowing workers to perform simultaneous tasks, and they are flexible to adapt to different communication contexts [19].
The acoustic noise of industrial scenarios has been a traditional problem for this technology. However, recent advances in noise cancelation and speech recognition have been proved to be robust enough for manufacturing environments [20].
The goal of gesture recognition is to use manpower as a direct input device, eliminating the need
for intermediate media and controlling the machine directly through defined gestures [21]. The
objective is to make machines understand the meaning of gestures of humans, based on technologies
such as Computer Vision or glove-based gesture recognition systems [
XR with voice recognition scenarios showcase example of BCC. [23] proposes a system for operators to carry out a certain task through the combination of XR technologies with voice interaction process control logic (Figure 2). The proposed work streamlines multiple input and output XR devices into the logical scheme of a voice recognition system, describing and validating a framework enhancing human–machine communication interfaces. The authors showcase two examples empowering operators. The first one focuses on an operator maintaining the gripper of a Universal Robot using HoloLens Glasses. As this operation requires the simultaneous use of both hands, voice based interaction suits its requirements. The second example focuses on the assembling of cables for electric panels with an Augmented Reality system combining different uses of the projections and voice-based responses. Moreover, aural responses are also projected for operators with hearing impairments.
[24] presents an optical inspection-guiding system for electronic board manufacturing. The system monitors in real time the mounting process of electronic components performed by an operator. It visually guides the operator through the mounting process while checking the correctness of its actions. Thus, mounting errors are reduced while operator comfort is enhanced. The interaction with the operator is based on a Computer Vision system to recognize the correct mounting of the board coupled with an AR system, extended with a voice recognition interface, projecting information into the real board and a screen, where additional data and controls are located (Figure 2).
This section presents some examples of the application of the previous technologies to create BCCs to generate Industry 5.0 human-centric workflows within AI-based services. While Industry 4.0 AI manufacturing services are mainly focused on optimizing operational efficiency, within Industry 5.0 AI services should optimize cost functions including more global footprints of the manufacturing processes. This requires an evolution of both AI models and industrial KPIs and metrics, such as OEE.
Furthermore, human-machine collaboration must be more flexible. CPS, robot, cobots and even AI models should be able to harmonize human interactions. This will lead to an Augmented Intelligence, which can be defined as the "synergistic technology of humans and computers" [25], merging Human Intelligence (HI) and Artificial Intelligence (AI) and understanding intelligence as a fundamentally distributed phenomenon [26]. AI based systems should evolve from rigid pattern recognition capabilities to manage less structured and more chaotic scenarios, more suitable to resemble the complexity of real problems. This requires the integration of rules, laws, ontologies or functionally equivalent technologies into AI models, which currently is a challenging open research area.
Focusing on BCCs, they are critical to enrich decision flows and to integrate humans into them. Moreover, they could improve explicability and interpretability of current AI algorithms. Interpretability has to do with how accurate a machine learning model can associate a cause to an effect. Explicability has to do with the ability of the parameters, often hidden in deep networks, to justify the results. Humans may ask about characteristics of AI algorithms before approving their output, applying previous technologies to lay a natural interface adapted to humans and the manufacturing environment.
The integration of human knowledge to improve, customize, tune and command AI algorithms requires a dialog where humans and machines assist each other at several tasks. This dialog will add value to human experience and knowledge, strengthening the human role of Industry 5.0. For example, interactive exploratory data analysis is one of the more general tasks where these channels may generate synergies. The interaction channel can be adapted to the context of humans and tasks, applying automatic data filters, custom visualizations, and data proposals and allowing humans to express queries in natural language avoiding training in query languages (for example, “show me in a bar chart the average temperature of the core temperature of the process for the last 20 production cycles”).
Data labelling and annotation is another common required task for several AI services be trained. Although profound advances are being developed in the field of automatic labelling and annotation, human intervention is still required to further improve the quality of the output. BCCs will allow humans to guide automatic labelling algorithms, improving the output in iterative workflows. Humans may select a subset of images or data (similar to current captcha systems), approve the output of the algorithm, or guide the labelling algorithm, while the system informs of the output and provides alternatives and suggestions of following steps.
The same workflow applies to interactive and active learning, where an AI algorithm generates iterative outputs that are increasingly improved by decisions and expertise from humans. The system automatically finds results and ask humans when it does not know the best next step or requires feedback and validation. This feedback and validation is integrated into the knowledge based on the algorithms, improving future results automatically. Furthermore, humans could ask algorithms about foundations of their decisions in order to understand and validate the reasons underneath them.
Regarding prescriptive analytics, besides integrating humans in the flow, these channels will empower humans to ask suitable models for prescriptions of further scenarios to foster quality databased decisions, or to identify potentially dangerous or relevant situations that may harm productivity.
New synergies between human and CPS are key aspects for generating collaborators instead of competitors, strengthening the human role of Industry 5.0. BCCs between humans and machines lay the foundation of these synergies. This paper has analyzed main visualization and interaction technologies to generate these BCCs: visual analytics; augmented, virtual and mixed reality; voice recognition; natural language processing and gesture recognition. Although the integration of these technologies into new generation interfaces is an open research area, examples of the individual use of the technologies within manufacturing scenarios already exist.
The successful design, development and integration of these BCCs poses relevant open research challenges for the future to revolutionize current manufacturing environments, specially related to the integration of humans into AI service and algorithm workflows.
[18] C. Flavián, S. Ibáñez-Sánchez, C. Orús, The impact of virtual, augmented and mixed reality technologies on the customer experience, Journal of Business Research 100 (2019) 547–560. doi: 10.1016/j.jbusres.2018.10.050 [19] A. del Pozo, L. Garcia-Sardiña, M. Serras, A. Gonzalez-Docasal, M. I. Torres, … I. Etxebeste, EKIN: Towards natural language interaction with industrial production machines, CEUR Workshop Proceedings 2968 (2021) 5-8. [20] R. Gaizauskas, Investigating spoken dialogue to support manufacturing processes, 2019. URL: https://connectedeverything.ac.uk/feasibility-studies/spoken-dialogue-manufacturing/ [21] O. K. Oyedotun, A. Khashman, Deep learning in vision-based static hand gesture recognition,
Neural Computing and Applications 28 (2017) 3941–3951. doi: 10.1007/s00521-016-2294-8. [22] L. Roda-Sanchez, T. Olivares, C. Garrido-Hidalgo, A. Fernández-Caballero, Gesture Control Wearables for Human-Machine Interaction in Industry 4.0, in: Lecture Notes in Computer Science, vol. 11487 LNCS, Springer-Verlag, London, 2019, pp. 99-108. [23] M. Serras, L. García-Sardiña, B. Simões, H. Álvarez, J. Arambarri, Dialogue Enhanced Extended Reality: Interactive System for the Operator 4.0, Applied Sciences 10 (2020) 3960. doi: 10.3390/app10113960. [24] M. Ojer, I. Serrano, F. A. Saiz, I. Barandiaran, I. Gill, D. Aguinaga, D. Alejandro, Real-time automatic optical system to assist operators in the assembling of electronic components, International Journal of Advanced Manufacturing Technology 107 (2020) 2261–2275. doi: 10.1007/s00170-020-05125-z [25] M. N. O. Sadiku, T. J. Ashaolu, A. Ajayi-Majebi, S. M. Musa, Augmented Intelligence,
International Journal of Scientific Advances 2 (2021) 772-776. doi: 10.51542/ijscia.v2i5.17 [26] J. Ito, Extended Intelligence, 2016. URL: https://pubpub.ito.com/pub/extendedintelligence/release/1