Industry 4.0 is a paradigm, centered around the emergence of cyber-physical objects, and offering a promise of enhanced efficiency through digital connectivity and Artificial Intelligence (AI). According to the European Commission, Industry 4.0, as currently conceived, "is not fit for purpose in a context of climate crisis and planetary emergency, nor does it address deep social tensions" [1].
Although nowadays the Industry 4.0 vision is not a reality, the European Union envisions the Industry 5.0 as a new paradigm evolving a production and consumption driven economic model into a more transformative view of growth that is focused on human progress and well-being based on reducing and shifting consumption to new forms of sustainable, circular and regenerative economic value creation and equitable prosperity. The objective is to "seek people-planet-prosperity, combining competitiveness and sustainability, rather than simply value extraction to benefit shareholders" [1].
One of the main objectives of Industry 5.0 is to bring back human workers to the factory floors, generating synergies combining the human brainpower and creativity with the automation and AI technologies of semi and/or fully autonomous machines [2]. This objective has not been successfully met by Industry 4.0. For example, [3] analyze the role of operators in Industry 4.0 and identify three main technology challenges to (i) support operators to perform process tasks, (ii) support operators to understand and make decisions, and (iii) to learn from the activity of the operators in order to be able to predict specific situations, optimize the process and better organize the Smart Factory.
A Bidirectional Communication Channel (BCC) is a requirement to generate these synergies, leading to effective human-in-the-loop systems. Human-in-the-loop is an anthropocentric mechanism, which allows a direct sharing or transfer of human skills to a subset of CPS control loops [4].
To effectively collaborate with humans, CPS must adequately understand human intention and desire. Moreover, humans must have the means to understand, analyze and trust predictions and actions from CPS. According to [5], this tight interaction between CPS and human requires (i) a rich unambiguous bidirectional information flow and (ii) a proper set of abstract interactive humanmachine interfaces (HMIs). Industry HMIs have evolved from basic light, buttons and levers to advanced graphical user interfaces (GUI) with touch screens. Moreover, they keep evolving to multimodal interfaces supporting new interaction channels [6].
For example, [5] introduce the natural human-machine interfaces (NHMI) as the interfaces reducing the technological barriers required for a rich interaction. They present an application of NHMI to integrate humans within the decision-making process of a cybernetic control loop of an assembly system with cobots. They analyze expertise transfer between humans and CPS, but they do not focus on decision making mechanism for CPS. This approach is shared by this paper, focusing on the visualization and interaction technologies and use cases, but not on the technologies to implement the use cases.
Focusing on HMI, [7] analyze a collaborative decision-making process where acceptance and adaptation of humans to the process is integrated. They provide an extensive literature review and distinguish Human-Computer Interaction (HCI), from HMI and Human-Technology Interaction (HTI). HTI encompasses the processes, actions and dialogues that a user engages in to interact with technology, whether it is a computer, machine or robot. They distinguish three potential HTI modes: (i) system first, where the human adapts to the actions of the system; (ii) human first, where the system adapts to instructions of the human; and (iii) hybrid, where human and autonomous system have equal responsibility with existing bidirectional communication channels. This paper tackles the hybrid mode as the one viable to successfully meet Industry 5.0 requirements.
Visualization and interaction technologies are the foundation of these BCCs for this new generation HMIs, connecting humans with machines, CPS, data and AI services. Although these technologies are already available to use within Industry 4.0, their effective integration is still an open research topic. This position paper reviews main currently available technologies to generate these BCCs, presenting some use cases and applications to encourage further research in this area.
This section introduces the most promising visualization and interaction technologies for the generation of BCCs for Industry 5.0. Although the integration of these technologies into new generation interfaces is an open research area, examples of the individual use of the technologies already exist. Figure 1 summarizes the contribution and the approach followed by this paper. At the left corner, different human profiles interacting with industrial systems are represented. In production systems, two main reference models for human activities have been identified: Human-inthe-Loop and Human-in-the-Mesh. The first one, traditionally related to operators, refers to situations in which the worker is directly participating in the process of products fabrication or assembling and its loop of control. The second one, traditionally related to managers and engineers, refers to situations where the worker participates in production planning and it loop of control [8]. In the future, new profiles may arise within Industry 5.0. Although both profiles perform different activities within the decision-making process phase, interaction and visualization technologies are key to establish the communication between humans, machines, CPS, data and AI services.
This BCC is composed by both Human-to-Machine and Machine-to-Human channels. Although not all the visualization and interaction technologies are suitable for both channels, their integration will foster the development of new BCCs empowering workers through the use of digital devices and endorsing a human centric approach to technology.
Visual Computing has already been identified as a key enabling technology for Industry 4.0. Visual Analytics, Augmented and Virtual Reality, Computer Vision, HCI and related technologies are central to many disruptive applications in a Smart Factory perspective [9,10]. Voice and gesture based interfaces have also been proposed to enhance communication between operators and CPS [11].
These technologies connect humans with the main elements of Industry 5.0 (Figure 1). The CPS represents the core communication element, as it merges the physical and the virtual world, connecting both to machines and to their digital representation, commonly named as digital twin. CPS capture data from the elements and feed AI services executed either at the edge, cloud or at both. Industry 5.0 aligns the objective of these AI services towards a sustainable, human-centric and resilient industry [12]. Reviewing the plethora of AI services for Industry 5.0 is out of the scope of this paper, interested readers are referred to existing updated industrial AI services reviews [13].
This paper focuses on the following visualization and interaction technologies: visual analytics; augmented, virtual and mixed reality; voice recognition; natural language processing and gesture recognition.
Visual Analytics (VA) combines the strength of human cognitive abilities with analysis methods to extract information from data. High-dimensional, real-time visualization allows the graphical expression of complex process variables at a fraction of the cost of full-scale digitalization [14]. VA combines machine intelligence with human intelligence to gain insight from the data to support informed decision-making. In a recent survey on the use of VA in manufacturing scenarios [15], the authors highlight the extensive need for professional domain specific knowledge and see human-inthe-loop analysis as one of the major ongoing key challenges of VA systems. Business Analytics is a special case of VA focused in the analysis of historical raw data in order to achieve useful and focused insights and a better understanding of the business performance areas [16].
Challenges faced by VA include (i) the integration of process relevant analytics and visualization, and (ii) the integration of aging workforce. Aligned with Industry 5.0 objectives, measures should be taken to ensure the ease of use and increased accessibility of VA, with minimal training and upskilling required to gain access to intuitive data visualization [14].
Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR) technologies are shaping new interaction environments. These technologies, which integrate physical and virtual objects at different levels, have received several definitions [17]. VR replicates an environment that simulates a physical presence in places in the real world or an imagined world, allowing the user to interact in that world. AR is a live, direct or indirect view of a physical, real-world environment whose elements are augmented (or supplemented) by computer-generated sensory input such as sound, video, graphics or GPS data. The real world content and the computer-generated content are not able to respond to each other. MR is the merging of real and virtual worlds to produce new environments and visualizations where physical and digital objects co-exist and interact in real time. eXtended Reality (XR) is composed by the full spectrum of technologies in the virtual-to-reality continuum, such as AR, VR and MR. XR is an overlay of synthetic content on the real world that is anchored to and interacts with the real world. The key characteristic of XR is that the synthetic and the real-world content are able to react to each other in real time. XR technology is paving the way to new forms of interaction that disrupt traditional desktop interaction. [18] describes related concepts and presents an agenda for future research.
Voice recognition and natural language processing systems have greatly increased their performance within the last years due to the integration of new models based on deep neural networks. Conversational assistants, such as general-purpose Apple Siri, Google Now, Microsoft Cortana or Amazon Alexa, have become the main example of this improvement, simplifying and making human-machine interaction more natural. Voice assistants could act as a central interaction technology for Industry 5.0. As they are natural for workers, they require minimal training, lowering the knowledge barrier for existing workforce to interact with CPS. Moreover, they are both eyes-free and hands-free, allowing workers to perform simultaneous tasks, and they are flexible to adapt to different communication contexts [19].
The acoustic noise of industrial scenarios has been a traditional problem for this technology. However, recent advances in noise cancelation and speech recognition have been proved to be robust enough for manufacturing environments [20].
The goal of gesture recognition is to use manpower as a direct input device, eliminating the need for intermediate media and controlling the machine directly through defined gestures [21]. The objective is to make machines understand the meaning of gestures of humans, based on technologies such as Computer Vision or glove-based gesture recognition systems [10,22]. Gesture recognition not only presents technological challenges, the definition of gestures in order to become robust against individual variations in the performance is an open research field.
XR with voice recognition scenarios showcase example of BCC. [23] proposes a system for operators to carry out a certain task through the combination of XR technologies with voice interaction process control logic (Figure 2). The proposed work streamlines multiple input and output XR devices into the logical scheme of a voice recognition system, describing and validating a framework enhancing human-machine communication interfaces. The authors showcase two examples empowering operators. The first one focuses on an operator maintaining the gripper of a Universal Robot using HoloLens Glasses. As this operation requires the simultaneous use of both hands, voice based interaction suits its requirements. The second example focuses on the assembling of cables for electric panels with an Augmented Reality system combining different uses of the projections and voice-based responses. Moreover, aural responses are also projected for operators with hearing impairments.
Examples for XR with voice recognition [24] presents an optical inspection-guiding system for electronic board manufacturing. The system monitors in real time the mounting process of electronic components performed by an operator. It visually guides the operator through the mounting process while checking the correctness of its actions. Thus, mounting errors are reduced while operator comfort is enhanced. The interaction with the operator is based on a Computer Vision system to recognize the correct mounting of the board coupled with an AR system, extended with a voice recognition interface, projecting information into the real board and a screen, where additional data and controls are located (Figure 2).
This section presents some examples of the application of the previous technologies to create BCCs to generate Industry 5.0 human-centric workflows within AI-based services. While Industry 4.0 AI manufacturing services are mainly focused on optimizing operational efficiency, within Industry 5.0 AI services should optimize cost functions including more global footprints of the manufacturing processes. This requires an evolution of both AI models and industrial KPIs and metrics, such as OEE.
Furthermore, human-machine collaboration must be more flexible. CPS, robot, cobots and even AI models should be able to harmonize human interactions. This will lead to an Augmented Intelligence, which can be defined as the "synergistic technology of humans and computers" [25], merging Human Intelligence (HI) and Artificial Intelligence (AI) and understanding intelligence as a fundamentally distributed phenomenon [26]. AI based systems should evolve from rigid pattern recognition capabilities to manage less structured and more chaotic scenarios, more suitable to resemble the complexity of real problems. This requires the integration of rules, laws, ontologies or functionally equivalent technologies into AI models, which currently is a challenging open research area.
Focusing on BCCs, they are critical to enrich decision flows and to integrate humans into them. Moreover, they could improve explicability and interpretability of current AI algorithms. Interpretability has to do with how accurate a machine learning model can associate a cause to an effect. Explicability has to do with the ability of the parameters, often hidden in deep networks, to justify the results. Humans may ask about characteristics of AI algorithms before approving their output, applying previous technologies to lay a natural interface adapted to humans and the manufacturing environment.
The integration of human knowledge to improve, customize, tune and command AI algorithms requires a dialog where humans and machines assist each other at several tasks. This dialog will add value to human experience and knowledge, strengthening the human role of Industry 5.0. For example, interactive exploratory data analysis is one of the more general tasks where these channels may generate synergies. The interaction channel can be adapted to the context of humans and tasks, applying automatic data filters, custom visualizations, and data proposals and allowing humans to express queries in natural language avoiding training in query languages (for example, "show me in a bar chart the average temperature of the core temperature of the process for the last 20 production cycles").
Data labelling and annotation is another common required task for several AI services be trained. Although profound advances are being developed in the field of automatic labelling and annotation, human intervention is still required to further improve the quality of the output. BCCs will allow humans to guide automatic labelling algorithms, improving the output in iterative workflows. Humans may select a subset of images or data (similar to current captcha systems), approve the output of the algorithm, or guide the labelling algorithm, while the system informs of the output and provides alternatives and suggestions of following steps.
The same workflow applies to interactive and active learning, where an AI algorithm generates iterative outputs that are increasingly improved by decisions and expertise from humans. The system automatically finds results and ask humans when it does not know the best next step or requires feedback and validation. This feedback and validation is integrated into the knowledge based on the algorithms, improving future results automatically. Furthermore, humans could ask algorithms about foundations of their decisions in order to understand and validate the reasons underneath them.
Regarding prescriptive analytics, besides integrating humans in the flow, these channels will empower humans to ask suitable models for prescriptions of further scenarios to foster quality databased decisions, or to identify potentially dangerous or relevant situations that may harm productivity.
New synergies between human and CPS are key aspects for generating collaborators instead of competitors, strengthening the human role of Industry 5.0. BCCs between humans and machines lay the foundation of these synergies. This paper has analyzed main visualization and interaction technologies to generate these BCCs: visual analytics; augmented, virtual and mixed reality; voice recognition; natural language processing and gesture recognition. Although the integration of these technologies into new generation interfaces is an open research area, examples of the individual use of the technologies within manufacturing scenarios already exist.
The successful design, development and integration of these BCCs poses relevant open research challenges for the future to revolutionize current manufacturing environments, specially related to the integration of humans into AI service and algorithm workflows.