=Paper=
{{Paper
|id=Vol-3214/WS5Paper5
|storemode=property
|title=Relevance of Visualization and Interaction Technologies for Industry 5.0
|pdfUrl=https://ceur-ws.org/Vol-3214/WS5Paper5.pdf
|volume=Vol-3214
|authors=Ander Garcia,Marco Quartulli,Igor G. Olaizola,Iñigo Barandiaran
|dblpUrl=https://dblp.org/rec/conf/iesa/GarciaQOB22
}}
==Relevance of Visualization and Interaction Technologies for Industry 5.0==
https://ceur-ws.org/Vol-3214/WS5Paper5.pdf
Relevance of Visualization and Interaction Technologies for
Industry 5.0
Ander Garcia1, Marco Quartulli1, Igor G. Olaizola1, and Iñigo Barandiaran1
1
Vicomtech Foundation, Basque Research and Technology Alliance (BRTA), Mikeletegi 57, 20009 Donostia-
San Sebastián (Spain)
Abstract
New synergies between human and Cyber Physical Systems (CPS) are key aspects for
generating collaborators instead of competitors, strengthening the human role of Industry 5.0.
Bidirectional Communication Channels (BCCs) between humans and machines lay the
foundation of these synergies within manufacturing environments. This position paper
reviews main currently available visualization and interaction technologies to connect data,
humans, CPS, Artificial Intelligence (AI) and machines, presenting a selection of use cases
and applications for AI services. The successful design, development and integration of these
bidirectional communication channels poses relevant open research challenges for the future
to revolutionize current manufacturing environments.
Keywords 1
Visual computing, Industry 4.0, Industry 5.0, AI, human-in-the-loop
1. Introduction
Industry 4.0 is a paradigm, centered around the emergence of cyber-physical objects, and offering
a promise of enhanced efficiency through digital connectivity and Artificial Intelligence (AI).
According to the European Commission, Industry 4.0, as currently conceived, “is not fit for purpose
in a context of climate crisis and planetary emergency, nor does it address deep social tensions” [1].
Although nowadays the Industry 4.0 vision is not a reality, the European Union envisions the
Industry 5.0 as a new paradigm evolving a production and consumption driven economic model into a
more transformative view of growth that is focused on human progress and well-being based on
reducing and shifting consumption to new forms of sustainable, circular and regenerative economic
value creation and equitable prosperity. The objective is to “seek people-planet-prosperity, combining
competitiveness and sustainability, rather than simply value extraction to benefit shareholders” [1].
One of the main objectives of Industry 5.0 is to bring back human workers to the factory floors,
generating synergies combining the human brainpower and creativity with the automation and AI
technologies of semi and/or fully autonomous machines [2]. This objective has not been successfully
met by Industry 4.0. For example, [3] analyze the role of operators in Industry 4.0 and identify three
main technology challenges to (i) support operators to perform process tasks, (ii) support operators to
understand and make decisions, and (iii) to learn from the activity of the operators in order to be able
to predict specific situations, optimize the process and better organize the Smart Factory.
A Bidirectional Communication Channel (BCC) is a requirement to generate these synergies,
leading to effective human-in-the-loop systems. Human-in-the-loop is an anthropocentric mechanism,
which allows a direct sharing or transfer of human skills to a subset of CPS control loops [4].
Proceedings of the Workshop of I-ESA’22, March 23–24, 2022, Valencia, Spain
EMAIL: agarcia@vicomtech.org (A. Garcia); mquartulli@vicomtech.org (M. Quartulli); iolaizola@vicomtech.org (I.G. Olaizola);
ibarandiaran@vicomtech.org (I. Barandiaran)
ORCID: 0000-0001-5596-2838 (A. Garcia); 0000-0001-5735-2072 (M. Quartulli); 0000-0002-9965-2038 (I.G. Olaizola); 0000-0002-8080-
5807 (I. Barandiaran)
© 2022 Copyright for this paper by its authors.
Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
Wor
Pr
ks
hop
oceedi
ngs
ht
I
tp:
//
ceur
-
SSN1613-
ws
.or
0073
g
CEUR Workshop Proceedings (CEUR-WS.org)
To effectively collaborate with humans, CPS must adequately understand human intention and
desire. Moreover, humans must have the means to understand, analyze and trust predictions and
actions from CPS. According to [5], this tight interaction between CPS and human requires (i) a rich
unambiguous bidirectional information flow and (ii) a proper set of abstract interactive human-
machine interfaces (HMIs). Industry HMIs have evolved from basic light, buttons and levers to
advanced graphical user interfaces (GUI) with touch screens. Moreover, they keep evolving to multi-
modal interfaces supporting new interaction channels [6].
For example, [5] introduce the natural human-machine interfaces (NHMI) as the interfaces
reducing the technological barriers required for a rich interaction. They present an application of
NHMI to integrate humans within the decision-making process of a cybernetic control loop of an
assembly system with cobots. They analyze expertise transfer between humans and CPS, but they do
not focus on decision making mechanism for CPS. This approach is shared by this paper, focusing on
the visualization and interaction technologies and use cases, but not on the technologies to implement
the use cases.
Focusing on HMI, [7] analyze a collaborative decision-making process where acceptance and
adaptation of humans to the process is integrated. They provide an extensive literature review and
distinguish Human-Computer Interaction (HCI), from HMI and Human-Technology Interaction
(HTI). HTI encompasses the processes, actions and dialogues that a user engages in to interact with
technology, whether it is a computer, machine or robot. They distinguish three potential HTI modes:
(i) system first, where the human adapts to the actions of the system; (ii) human first, where the
system adapts to instructions of the human; and (iii) hybrid, where human and autonomous system
have equal responsibility with existing bidirectional communication channels. This paper tackles the
hybrid mode as the one viable to successfully meet Industry 5.0 requirements.
Visualization and interaction technologies are the foundation of these BCCs for this new
generation HMIs, connecting humans with machines, CPS, data and AI services. Although these
technologies are already available to use within Industry 4.0, their effective integration is still an open
research topic. This position paper reviews main currently available technologies to generate these
BCCs, presenting some use cases and applications to encourage further research in this area.
2. Visualization and interaction technologies for Industry 5.0
This section introduces the most promising visualization and interaction technologies for the
generation of BCCs for Industry 5.0. Although the integration of these technologies into new
generation interfaces is an open research area, examples of the individual use of the technologies
already exist. Figure 1 summarizes the contribution and the approach followed by this paper.
Figure 1: Visualization and Interaction technologies for Industry 5.0
At the left corner, different human profiles interacting with industrial systems are represented. In
production systems, two main reference models for human activities have been identified: Human-in-
the-Loop and Human-in-the-Mesh. The first one, traditionally related to operators, refers to situations
in which the worker is directly participating in the process of products fabrication or assembling and
its loop of control. The second one, traditionally related to managers and engineers, refers to
situations where the worker participates in production planning and it loop of control [8]. In the
future, new profiles may arise within Industry 5.0. Although both profiles perform different activities
within the decision-making process phase, interaction and visualization technologies are key to
establish the communication between humans, machines, CPS, data and AI services.
This BCC is composed by both Human-to-Machine and Machine-to-Human channels. Although
not all the visualization and interaction technologies are suitable for both channels, their integration
will foster the development of new BCCs empowering workers through the use of digital devices and
endorsing a human centric approach to technology.
Visual Computing has already been identified as a key enabling technology for Industry 4.0.
Visual Analytics, Augmented and Virtual Reality, Computer Vision, HCI and related technologies are
central to many disruptive applications in a Smart Factory perspective [9, 10]. Voice and gesture
based interfaces have also been proposed to enhance communication between operators and CPS [11].
These technologies connect humans with the main elements of Industry 5.0 (Figure 1). The CPS
represents the core communication element, as it merges the physical and the virtual world,
connecting both to machines and to their digital representation, commonly named as digital twin. CPS
capture data from the elements and feed AI services executed either at the edge, cloud or at both.
Industry 5.0 aligns the objective of these AI services towards a sustainable, human-centric and
resilient industry [12]. Reviewing the plethora of AI services for Industry 5.0 is out of the scope of
this paper, interested readers are referred to existing updated industrial AI services reviews [13].
This paper focuses on the following visualization and interaction technologies: visual analytics;
augmented, virtual and mixed reality; voice recognition; natural language processing and gesture
recognition.
Visual Analytics (VA) combines the strength of human cognitive abilities with analysis methods
to extract information from data. High-dimensional, real-time visualization allows the graphical
expression of complex process variables at a fraction of the cost of full-scale digitalization [14]. VA
combines machine intelligence with human intelligence to gain insight from the data to support
informed decision-making. In a recent survey on the use of VA in manufacturing scenarios [15], the
authors highlight the extensive need for professional domain specific knowledge and see human-in-
the-loop analysis as one of the major ongoing key challenges of VA systems. Business Analytics is a
special case of VA focused in the analysis of historical raw data in order to achieve useful and
focused insights and a better understanding of the business performance areas [16].
Challenges faced by VA include (i) the integration of process relevant analytics and visualization,
and (ii) the integration of aging workforce. Aligned with Industry 5.0 objectives, measures should be
taken to ensure the ease of use and increased accessibility of VA, with minimal training and
upskilling required to gain access to intuitive data visualization [14].
Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR) technologies are
shaping new interaction environments. These technologies, which integrate physical and virtual
objects at different levels, have received several definitions [17]. VR replicates an environment that
simulates a physical presence in places in the real world or an imagined world, allowing the user to
interact in that world. AR is a live, direct or indirect view of a physical, real-world environment
whose elements are augmented (or supplemented) by computer-generated sensory input such as
sound, video, graphics or GPS data. The real world content and the computer-generated content are
not able to respond to each other. MR is the merging of real and virtual worlds to produce new
environments and visualizations where physical and digital objects co-exist and interact in real time.
eXtended Reality (XR) is composed by the full spectrum of technologies in the virtual-to-reality
continuum, such as AR, VR and MR. XR is an overlay of synthetic content on the real world that is
anchored to and interacts with the real world. The key characteristic of XR is that the synthetic and
the real-world content are able to react to each other in real time. XR technology is paving the way to
new forms of interaction that disrupt traditional desktop interaction. [18] describes related concepts
and presents an agenda for future research.
Voice recognition and natural language processing systems have greatly increased their
performance within the last years due to the integration of new models based on deep neural
networks. Conversational assistants, such as general-purpose Apple Siri, Google Now, Microsoft
Cortana or Amazon Alexa, have become the main example of this improvement, simplifying and
making human-machine interaction more natural. Voice assistants could act as a central interaction
technology for Industry 5.0. As they are natural for workers, they require minimal training, lowering
the knowledge barrier for existing workforce to interact with CPS. Moreover, they are both eyes-free
and hands-free, allowing workers to perform simultaneous tasks, and they are flexible to adapt to
different communication contexts [19].
The acoustic noise of industrial scenarios has been a traditional problem for this technology.
However, recent advances in noise cancelation and speech recognition have been proved to be robust
enough for manufacturing environments [20].
The goal of gesture recognition is to use manpower as a direct input device, eliminating the need
for intermediate media and controlling the machine directly through defined gestures [21]. The
objective is to make machines understand the meaning of gestures of humans, based on technologies
such as Computer Vision or glove-based gesture recognition systems [10, 22]. Gesture recognition not
only presents technological challenges, the definition of gestures in order to become robust against
individual variations in the performance is an open research field.
XR with voice recognition scenarios showcase example of BCC. [23] proposes a system for
operators to carry out a certain task through the combination of XR technologies with voice
interaction process control logic (Figure 2). The proposed work streamlines multiple input and output
XR devices into the logical scheme of a voice recognition system, describing and validating a
framework enhancing human–machine communication interfaces. The authors showcase two
examples empowering operators. The first one focuses on an operator maintaining the gripper of a
Universal Robot using HoloLens Glasses. As this operation requires the simultaneous use of both
hands, voice based interaction suits its requirements. The second example focuses on the assembling
of cables for electric panels with an Augmented Reality system combining different uses of the
projections and voice-based responses. Moreover, aural responses are also projected for operators
with hearing impairments.
Figure 2: Examples for XR with voice recognition
[24] presents an optical inspection-guiding system for electronic board manufacturing. The system
monitors in real time the mounting process of electronic components performed by an operator. It
visually guides the operator through the mounting process while checking the correctness of its
actions. Thus, mounting errors are reduced while operator comfort is enhanced. The interaction with
the operator is based on a Computer Vision system to recognize the correct mounting of the board
coupled with an AR system, extended with a voice recognition interface, projecting information into
the real board and a screen, where additional data and controls are located (Figure 2).
3. Example of applications of BCCs for AI manufacturing services
This section presents some examples of the application of the previous technologies to create
BCCs to generate Industry 5.0 human-centric workflows within AI-based services. While Industry 4.0
AI manufacturing services are mainly focused on optimizing operational efficiency, within Industry
5.0 AI services should optimize cost functions including more global footprints of the manufacturing
processes. This requires an evolution of both AI models and industrial KPIs and metrics, such as
OEE.
Furthermore, human-machine collaboration must be more flexible. CPS, robot, cobots and even AI
models should be able to harmonize human interactions. This will lead to an Augmented Intelligence,
which can be defined as the "synergistic technology of humans and computers" [25], merging Human
Intelligence (HI) and Artificial Intelligence (AI) and understanding intelligence as a fundamentally
distributed phenomenon [26]. AI based systems should evolve from rigid pattern recognition
capabilities to manage less structured and more chaotic scenarios, more suitable to resemble the
complexity of real problems. This requires the integration of rules, laws, ontologies or functionally
equivalent technologies into AI models, which currently is a challenging open research area.
Focusing on BCCs, they are critical to enrich decision flows and to integrate humans into them.
Moreover, they could improve explicability and interpretability of current AI algorithms.
Interpretability has to do with how accurate a machine learning model can associate a cause to an
effect. Explicability has to do with the ability of the parameters, often hidden in deep networks, to
justify the results. Humans may ask about characteristics of AI algorithms before approving their
output, applying previous technologies to lay a natural interface adapted to humans and the
manufacturing environment.
The integration of human knowledge to improve, customize, tune and command AI algorithms
requires a dialog where humans and machines assist each other at several tasks. This dialog will add
value to human experience and knowledge, strengthening the human role of Industry 5.0. For
example, interactive exploratory data analysis is one of the more general tasks where these channels
may generate synergies. The interaction channel can be adapted to the context of humans and tasks,
applying automatic data filters, custom visualizations, and data proposals and allowing humans to
express queries in natural language avoiding training in query languages (for example, “show me in a
bar chart the average temperature of the core temperature of the process for the last 20 production
cycles”).
Data labelling and annotation is another common required task for several AI services be trained.
Although profound advances are being developed in the field of automatic labelling and annotation,
human intervention is still required to further improve the quality of the output. BCCs will allow
humans to guide automatic labelling algorithms, improving the output in iterative workflows. Humans
may select a subset of images or data (similar to current captcha systems), approve the output of the
algorithm, or guide the labelling algorithm, while the system informs of the output and provides
alternatives and suggestions of following steps.
The same workflow applies to interactive and active learning, where an AI algorithm generates
iterative outputs that are increasingly improved by decisions and expertise from humans. The system
automatically finds results and ask humans when it does not know the best next step or requires
feedback and validation. This feedback and validation is integrated into the knowledge based on the
algorithms, improving future results automatically. Furthermore, humans could ask algorithms about
foundations of their decisions in order to understand and validate the reasons underneath them.
Regarding prescriptive analytics, besides integrating humans in the flow, these channels will
empower humans to ask suitable models for prescriptions of further scenarios to foster quality data-
based decisions, or to identify potentially dangerous or relevant situations that may harm productivity.
4. Conclusions
New synergies between human and CPS are key aspects for generating collaborators instead of
competitors, strengthening the human role of Industry 5.0. BCCs between humans and machines lay
the foundation of these synergies. This paper has analyzed main visualization and interaction
technologies to generate these BCCs: visual analytics; augmented, virtual and mixed reality; voice
recognition; natural language processing and gesture recognition. Although the integration of these
technologies into new generation interfaces is an open research area, examples of the individual use of
the technologies within manufacturing scenarios already exist.
The successful design, development and integration of these BCCs poses relevant open research
challenges for the future to revolutionize current manufacturing environments, specially related to the
integration of humans into AI service and algorithm workflows.
5. References
[1] European Commission, Industry 5.0: A Transformative Vision for Europe, Governing Systemic
Transformations towards a Sustainable Industry, 2022. URL:
https://ec.europa.eu/info/publications/industry-50-transformative-vision-europe_en
[2] S. Nahavandi, Industry 5.0 - A Human-Centric Solution, Sustainability 11 (2019) 4371. doi:
10.3390/su11164371.
[3] J. Posada, M. Zorrilla, A. Dominguez, B. Simoes, P. Eisert, … M. Guevara, Graphics and Media
Technologies for Operators in Industry 4.0, IEEE Computer Graphics and Applications 38
(2018) 119–132. doi: 10.1109/MCG.2018.053491736.
[4] M. Gaham, B. Bouzouia, N. Achour, Human-in-the-Loop Cyber-Physical Production Systems
Control Human-in-the-Loop Cyber-Physical Production Systems Control (HiLCP2sC), Studies
in Computational Intelligence 594 (2015) 315-325.
[5] M. A. Ruiz Garcia, R. Rojas, L. Gualtieri, E. Rauch, D. Matt, A human-in-the-loop cyber-
physical system for collaborative assembly in smart manufacturing, Procedia CIRP 81 (2019)
600–605. doi: 10.1016/j.procir.2019.03.162
[6] S. Oviatt, P. Cohen, Perceptual User Interfaces: Multimodal Interfaces that Process What Comes
Naturally, Communications of the ACM 43 (2000) 45–53. doi: 10.1145/330534.330538.
[7] J. Coetzer, R. B. Kuriakose, H. J. Vermaak, Collaborative Decision-Making for Human-
Technology Interaction-A Case Study Using an Automated Water Bottling Plant, Journal of
Physics: Conference Series 1577 (2020) 012024.
[8] P. Fantini, P. Leitao, J. Barbosa, M. Taisch, Symbiotic Integration of Human Activities in Cyber-
Physical Systems, IFAC-PapersOnLine 52 (2019) 133-138. doi: 10.1016/j.ifacol.2019.12.124.
[9] J. Posada, C. Toro, I. Barandiaran, D. Oyarzun, P. Eisert, Visual computing as key enabling
technology for Industry 4.0 & Industrial Internet, IEEE Computer Graphics and Applications 35
(2015) 26-40. doi: 10.1109/MCG.2015.45.
[10] J. Posada, I. Barandiaran, J. R. Sanchez, D. Mejia-Parra, A. Moreno, M. Ojer, O. Ruiz-Salguero,
Computer graphics and visual computing use cases for Industry 4.0 and Operator 4.0,
International Journal for Simulation and Multidisciplinary Design Optimization 12 (2021) 4-9.
doi: 10.1051/smdo/2021026
[11] F. Longo, A. Padovano, Voice-enabled Assistants of the Operator 4.0 in the Social Smart
Factory: Prospective role and challenges for an advanced human–machine interaction,
Manufacturing Letters 26 (2020) 12–16. doi: 10.1016/j.mfglet.2020.09.001.
[12] M. Breque, L. De Nul, A. Petrides, Industry 5.0 - Towards a sustainable, human-centric and
resilient European industry, 2021. URL: https://ec.europa.eu/info/news/industry-50-towards-
more-sustainable-resilient-and-human-centric-industry-2021-jan-07_en
[13] R. S. Peres, X. Jia, J. Lee, K. Sun, A. W. Colombo, J. Barata, Industrial Artificial Intelligence in
Industry 4.0 -Systematic Review, Challenges and Outlook, IEEE Access 8 (2020) 220121–
220139. doi: 10.1109/ACCESS.2020.3042874.
[14] L. Allen, J. Atkinson, D. Jayasundara, J. Cordiner, P. Z. Moghadam, Data visualization for
Industry 4.0: A stepping-stone toward a digital future, bridging the gap between academia and
industry, Patterns 2 (2021) 100266. doi: 10.1016/j.patter.2021.100266
[15] F. Zhou, X. Lin, C. Liu, Y. Zhao, P. Xu, L. Ren, T. Xue, L. Ren, A survey of visualization for
smart manufacturing, Journal of Visualization 22 (2019) 419-435. doi: 10.1007/s12650-018-
0530-2.
[16] A. J. Silva, P. Cortez, C. Pereira, A. Pilastri, Business analytics in Industry 4.0: A systematic
review, Expert Systems 38 (2021) e12741. doi: 10.1111/exsy.12741.
[17] T. Williams, D. Szafir, T. Chakraborti, H. Ben Amor, Virtual, Augmented, and Mixed Reality for
Human-Robot Interaction, in: ACM/IEEE International Conference on Human-Robot
Interaction, IEEE, New York, 2018, pp. 671-672. doi: 10.1109/HRI.2019.8673207.
[18] C. Flavián, S. Ibáñez-Sánchez, C. Orús, The impact of virtual, augmented and mixed reality
technologies on the customer experience, Journal of Business Research 100 (2019) 547–560. doi:
10.1016/j.jbusres.2018.10.050
[19] A. del Pozo, L. Garcia-Sardiña, M. Serras, A. Gonzalez-Docasal, M. I. Torres, … I. Etxebeste,
EKIN: Towards natural language interaction with industrial production machines, CEUR
Workshop Proceedings 2968 (2021) 5-8.
[20] R. Gaizauskas, Investigating spoken dialogue to support manufacturing processes, 2019. URL:
https://connectedeverything.ac.uk/feasibility-studies/spoken-dialogue-manufacturing/
[21] O. K. Oyedotun, A. Khashman, Deep learning in vision-based static hand gesture recognition,
Neural Computing and Applications 28 (2017) 3941–3951. doi: 10.1007/s00521-016-2294-8.
[22] L. Roda-Sanchez, T. Olivares, C. Garrido-Hidalgo, A. Fernández-Caballero, Gesture Control
Wearables for Human-Machine Interaction in Industry 4.0, in: Lecture Notes in Computer
Science, vol. 11487 LNCS, Springer-Verlag, London, 2019, pp. 99-108.
[23] M. Serras, L. García-Sardiña, B. Simões, H. Álvarez, J. Arambarri, Dialogue Enhanced Extended
Reality: Interactive System for the Operator 4.0, Applied Sciences 10 (2020) 3960. doi:
10.3390/app10113960.
[24] M. Ojer, I. Serrano, F. A. Saiz, I. Barandiaran, I. Gill, D. Aguinaga, D. Alejandro, Real-time
automatic optical system to assist operators in the assembling of electronic components,
International Journal of Advanced Manufacturing Technology 107 (2020) 2261–2275. doi:
10.1007/s00170-020-05125-z
[25] M. N. O. Sadiku, T. J. Ashaolu, A. Ajayi-Majebi, S. M. Musa, Augmented Intelligence,
International Journal of Scientific Advances 2 (2021) 772-776. doi: 10.51542/ijscia.v2i5.17
[26] J. Ito, Extended Intelligence, 2016. URL: https://pubpub.ito.com/pub/extended-
intelligence/release/1