A viability and a rationale for implementation of digital twins with augmented reality interface for further development of the smart manufacturing ecosystem is discussed. Different aspects of the problem of design and implementation of digital twins for industrial applications are considered. An approach for constructing the secure-by-design augmented reality-enhanced interfaces for digital twins is proposed. Benefits and cautions for use of augmented reality-enhanced digital twins in Industry 4.0 and prospects for Industry 5.0 are discussed.
Digital twins are integrated nowadays into various spheres [
At the same time, in today's digital era, it is essential to take necessary measures and use appropriate technologies to ensure information security in every smart domain due to the prevalence of user privacy issues. Information security is critical for businesses and individuals as it prevents data breaches that can result in financial losses for organizations, frauds, extortion and identity thefts for individuals.
Transition to a more digitized industrial value chains is to have important outcomes for sustainable development, increase the energy efficiency of energy-intensive industrial processes and contribute to achieving the climate neutrality goal, using the creative potential of human enhanced by abilities of AI-based information systems. Despite the opportunities presented by digitalization and Industry 4.0 to 5.0 transition, which includes efficient use of IoT components and energy-efficient solutions that reduce pollution, there remain various aspects that can be integrated into these systems in order to further improve process operations. Important direction for the utilization of digital twin solutions is to optimize manufacturing lines and technologies specifically for the energy-intensive processes, especially in spatially distributed manufacturing chains. Furthermore, making the utmost use of raw materials and energy can aid in decreasing waste-generation, amplifying access to recycled materials, lowering the levels of energy consumption and greenhouse gas-emissions and leading to costreductions.
Modern production lines are multifunctional physical systems consisting of intelligent machines,
materials, products and containing a large number of multi-level connections between various
elements. In the process of digital design of such systems, a kind of division is often carried out
digital models of various kinds are arranged separately in the digital space, while physical products
and production processes exist in the physical space. A study of the development process of such
smart factories indicates [
According to the widely used definition by the National Institute of Standards and Technology,
Smart Manufacturing systems have to be collaborative manufacturing entities able to respond in real
time to condition and demand changes. Such communication of production system units is also a
characteristic feature of digital twins, which are digital representations of physical elements, usually
using IoT and sensor data to monitor operations, control physical elements and support
decisionmaking processes [
To acquire data, analyze and optimize energy consumption of a smart manufacturing production, for which material flows, data flows and particular processes are shown in Figure 1, design and deployment of an analytics platform is absolutely fundamental. It is crucial to put in place data management systems for the purpose of storing and managing energy data scenarios, consumption and production process parameters. By gathering data, identification of the most energy-consuming aspects of particular processes will be possible, which will then facilitate the integration of new digital technologies, distributed procedure management and data-powered optimization. The power consumption by equipment and processes can differ substantially due to various specific factors including the type of machinery, the conditions under which it operates, and the schedule of production. In order to determine the energy-intensive processes or equipment, it is necessary to gather data on energy consumption, production and efficiency, using specifically selected smart sensors, meters and other monitoring devices which will be linked to manufacturing machinery.
There are various possible energy data that can be collected from smart manufacturing line components, namely Physical material data / visual material data, Process control data / machine data, Environmental data, Synthetic and measured operational data.
Classification of these data types is proposed in Figure 2 for a generic smart manufacturing facility
using different types of raw materials and multiple sources to meet the energy demands in compliance
with regulations and policies of green and digital transition [
The power consumption by each individual machine in the production line is one of the crucial data points to gather. Modern machinery can have IoT sensors to measure their energy consumption in real-time and this information can be utilized to identify units and processes where the equipment consumes more energy than is required and to optimize energy consumption as a whole as well as to reduce waste heat production. With the help of digital twins, the industrial data platform can track the manufacturing line's energy usage even if it is distributed among several different physical sites.
There are a few sorts of environmental data that need to be gathered. Some smart manufacturing processes (such as 3D printing and molding of plastic materials) call for precise temperature management, thus keeping track of temperature information is essential to both maintaining highquality output and maximizing energy efficiency. Dataflow from the temperature sensors of the machine's heating and cooling systems can be used to both feed a digital twin that simulates the thermal behavior of the machine to predict potential problems and optimize the manufacturing processes and maintenance as well as to trigger edge-system controls in case of overheating.
Algorithms can improve the machine's performance to consume less energy by assessing data on the operating parameters and the amount of material being used. Vibration sensors on machine parts can make it possible to spot potential concerns before they become serious ones. Likewise, information regarding hydraulic system pressure will be gathered by sensors and used to modify energy usage as necessary. Through the machine's digital twin, information on the quantity of parts produced, cycle duration and downtime will be gathered and used to improve energy use.
Another benefit from digital twin implementation in smart manufacturing is predictive maintenance that can be used to maximize equipment time in service. The information gathered through sensors regarding the machine's state, encompassing data points such as vibration and temperature, allows for the prediction of maintenance needs, detecting causes of unanticipated interruptions to operations. By utilizing data analytics to schedule maintenance tasks, the workflow can be optimized for better performance, decreased energy consumption, and improved efficiency.
By implementing a specialized intelligent industrial data platform, it is possible to guarantee that the consistent and accurate data obtained from sensors will be utilized during subsequent analysis and modeling. The appropriate data governance practices must be used when gathering, retaining, and manipulating data related to industrial procedures. Properly accomplished data engineering (collecting, storing and preparing data) is an essential prerequisite for obtaining insight from the harvested data. Special attention is to be given to information security and the protection of privacy. This involves guaranteeing that data remains confidential, unaltered, and accessible, while also preventing unauthorized access, manipulation, and theft of data.
For an exemplary model of the smart manufacturing processes and units, specified by Figures 2 and 3, it is necessary to develop a model that represents the current state of the live production line through links to live data streams from the manufacturing floor and enterprise management or digital twin platform streams and provide options for decision-making based on mathematical models that allow characterizing both the resource consumption and process peculiarities. To identify the most effective and viable solution, several energy management scenarios can be simulated using the digital twin. A virtual copy of the smart manufacturing facility can enable real-time monitoring and analysis of the facility's performance and energy usage to optimize the process, resulting in a life cycle that is more sustainable.
Based on the data obtained from a set of embedded sensors and external data generators, a reference matrix would be developed consisting of the set of minimum parameters needed to mathematically describe each process and its dependencies, limits and boundary conditions. Target function built on this dataset may serve as a tool for optimization of the process and effectively the entire system. Optimization schema shown in Figure 4 illustrates the general approach which may help resolve the optimization problem in an efficient way. In the right panel, the business-oriented schematization is given. Inputs x and outputs y are the quantified characteristics of the material and information flows which the manufacturing system receives from the external entities and sends to the same or different external entities, respectively. Optimal control signal u* is the solution of the optimization problem to fulfill the goals set by particular smart manufacturing use case. This optimal u* is to be obtained following the procedure described in this section (the examples for particular usecases will be presented elsewhere), to improve the initial control signal u based merely on the model description of the process and standard procedures. The distinctive feature of the proposed approach is the model refinement being the integral part of the optimization problem solving.
Solving the general optimization problem is quite complicated as it comprises functional relations of different types for numerous variables of various nature. However, the equations describing a particular process contain only variables of ith subsystem and this feature allows a decomposition (see separate subsystems (for the detailed discussion on this approach applicability see [13, 14]). This approach also significantly reduces requirements for processing power used by the optimization engine. At the same time, if different partial optimization problems are self-contained, which may be the case of the distributed manufacturing system, the optimal solution for k incoherent with KPIs of the entire system, therefore no decision making may be done and no correcting actions may be taken before the full optimization solution is found for the entire system. th subsystem may be
Taking into account the measured process state vector ⃗ = ( 1 2 ⋯ ), control vector ⃗⃗ = ( 1 2 ⋯
)and disturbances, the model choice is performed and the chosen model is refined. The initial control signal u is chosen on basis of the following considerations. First, the dimensionality of the state vector on the each particular subsystem does not exceed the dimensionality of the control vector for this subsystem. Moreover, particular is constructed according to hardware specification for the n
th smart manufacturing system component, for example, as binary code or analog signal. For a complex system, in which every compact manufacturing process (ith subsystem) is characterized by its own control vector ⃗⃗ , input vector ⃗ , output vector ⃗ and transformation matrix , we consider the problem ⃗⃗⃗⃗ = ⃗⃗⃗⃗⃗
) = 0, ) = 0, = ∑ ⃗⃗⃗⃗ − ⃗⃗⃗⃗ = 0 (1) (2) (3) (4) (5) (6) (7) is to be solved and values ⃗ , ⃗⃗ , are corrected with use of the condition
⃗ , ⃗ are to be determined. In the next stage the outputs vectors ⃗ at conditions Lagrange polynomial for the above problem formulation is =1 ⃗⃗⃗⃗ =
∑ =1 ⃗⃗⃗⃗, ⃗⃗⃗⃗ = ⃗⃗⃗ (⃗⃗⃗⃗, ⃗⃗⃗⃗), = 1,2 ⋯ .
( ⃗ , ⃗ , ⃗⃗ , ⃗ , ⃗ ) = ∑ =1 ( ( ⃗ , ⃗⃗ )+ ⃗⃗⃗⃗ (∑ =1 ⃗⃗⃗⃗ − ⃗⃗⃗⃗) + ⃗⃗⃗⃗ (⃗⃗⃗ (⃗⃗⃗⃗, ⃗⃗⃗⃗)− ⃗⃗⃗⃗)).
Solution correction is performed on a higher level of the optimization, which corresponds to the overall system
management. Coordination of the solutions obtained for separate subsystems is to operate the output vectors ⃗ .Therefore, in the first stage the calculation of output vectors ⃗1, ⃗2, … ⃗ is to be done. In the second stage, the system of equations and the system of equation (4)-(7) is solved with the corrected parameters iteratively. The condition for interruption of the iterative procedure may be chosen as
When the acceptable solution is obtained with the desired tolerance, the procedure, which is wellsuited for on-line optimization, is interrupted and the controls are enforced. Worth noting, the described correction procedure improves the intermediate solution for a defined target function, which itself may be a subject for correction. Even if the best solution is not reached in certain iteration, the current suboptimal solution is an improvement over the initial control signal u.
Augmented Reality is a technology that can bring significant change in the growth of an organization. When combined with Artificial Intelligence (AI) and the Internet of Things, AR opens new possibilities in product manufacturing, maintenance, support, and more. In Smart Manufacturing, AR can allow production managers to view production KPIs and have an intra-factory overview of workstations and production lines in real-time for monitoring, identifying, analyzing, diagnosing and resolving problems and flaws. AR can also be experienced via wearable smart glasses, or a mobile phone or tablet with a camera. The device may use computer-generated virtual objects to assist users in performing complex tasks and getting real-time insights for informed decision-making.
As a relatively new and rapidly developing information technology, AR is a powerful tool to facilitate the interaction and, to some extent, the merging of physical and virtual space objects, providing a variety of production services through the widespread adoption of digital twins. AR also provides much better effect of immersion in the industrial environment and a more natural way of interaction for the subjects of the production process. Let us point out that the objects of physical space and the applied level of extended reality technology can be quite organically linked through virtual space. However, one of the bottlenecks and problems faced by the technology of digital twins in production is the proper implementation of the full range of interaction between the physical space and the virtual twin [15]. That is why augmented reality plays a special role at the current stage of supporting the practical implementation of industrial processes. The combination of AR and digital twins can improve the performance of industrial systems in different areas and at different stages of their life cycle, including design, manufacturing, distribution, installation, active operational use, service and end-of-life. Depending on the goals of the industrial process, one can focus [16] on different levels of overlap and mutual influence between the real object and its counterpart. The concept of a passive virtual twin refers to the transfer of physical data into the virtual space for the purpose of observation, i.e. the implementation of basic functions of status monitoring and alerting based on sensor data. Compared to a traditional web-oriented digital twin, this process can be significantly improved due to the specifics of AR devices. Unlike the virtual twin, its hybrid subspecies focuses on virtual and physical analysis and feedback, which includes the processing of contextual information. After collecting data from the physical space, real-time data analysis must be performed, and this process includes modeling, prediction, diagnosis and optimization, as well as feedback from the analysis results from the virtual world to the physical world. AR support significantly enriches on-site data analysis by adding object detection, scene capture and processing of cyber-physical interaction (for example, with the help of the Microsoft HoloLens 2 AR headset, the workspace itself will be perceived much more fully). And finally, the cognitive twin has the most powerful high-level toolkit, as it allows you to combine human intelligence and machine computing. There is an opportunity to dynamically solve more complex and unpredictable situations with the help of advanced computing capabilities, to organize a creative process (design, interaction with robotised processes, machine learning, etc.).
For example, in work [17] it is noted that the modeling of the assembly of multi-element products is considered one of the key technologies in the process of designing and manufacturing complex systems. Note that AR-based digital assembly technology is used to implement the overlay of an additional information layer, perception of the assembly scene, navigation of assembly operations, joint design of the assembly process, etc. A digital assembly model based on a digital twin should realistically simulate the assembly behavior of physical objects in a real environment. Through the interaction between the virtual assembly objects and the real assembly environment, the quality of the assembly design is effectively improved.
The approach of digital twins has proven itself efficient for interaction with individual elements of production lines. For example, implementing [18] such a system on a CNC milling machine with remote process control, where control delay and virtual processing accuracy are monitored, can be applied as an important part of smart manufacturing, having a high potential for application on various industrial machines and smart systems. Augmented reality approaches are actively used to optimize control of robotic systems [19, 20] in industrial production.
Augmented reality-enhanced digital twins will facilitate human-computer interaction and make it more natural and personalized within Industry 5.0 practices. As metrics to be included into the information layer of the digital twin for the human operator to make timely informed decisions, the characteristics of manufacturing processes performance as well as critical physical parameters, predictions of events, risks estimates are the most relevant candidates.
The workflow for developing AR tools has already been tested in educational use-cases [
It is important to properly design and develop a security layer for the Industrial Digital Twin in cloud/edge environments. Addressing various aspects of information security and cybersecurity threats is required for assuring protection of the IDT and IoT devices from malicious attacks. Information, collected by the digital twins and processed by the industrial data platform, represents the valuable business asset and therefore is to be the subject of a thorough analysis in order to be appropriately secured. To address security concerns during the IDT development, deployment and use, the following aspects have to be considered.
Since the IDT itself and as being a part of Cyber Physical Systems operates with sensitive data and privacy data, the best security practice compliant with industrial standards and regulations should be followed by default. Secure-by-design principle for developing IDT implies security requirements to be identified, which is one of the most important stages of the system development life cycle that allow the engineers to develop a quality, cost effective and secure system. Among approaches to identify security requirements within information and cyber security domains there is threat modeling that allows to identify security needs, locate threats and vulnerabilities, score their impact and severity, and prioritize solutions. It can be applied to a broad range of systems, including software, networks, distributed systems, IoT and industrial processes. To identify and describe potential threats and vulnerabilities to the IDT and to individual’s personal data, the STRIDE [21] and LINDUNN [22] threat modeling methodologies can be leveraged. Based on the IDT architecture, its applications and technologies analyzed in [23] we developed the general data flow diagram and the threat model shown in Figure 7.
The potential threats to the IDT and the data being processed within the Smart Manufacturing facility have been analyzed. The corresponding threat descriptions and mitigation actions have been systematized in Table 1. The proposed countermeasures will help engineers and security specialists to reduce the time and costs while designing or upgrading the IDT platform and its components.
Privacy threat modeling within the IDT development is a process of identifying and assessing potential threats to personal information. It helps organizations involved in Smart Manufacturing as well as individuals to develop strategies to mitigate these threats and protect data privacy. Since the privacy threat modeling requires specific inputs of certain manufacturing implementations it will be analyzed in more detail in a separate research.
Shift-left security approach adoption for the IDT development and usage will help to ensure the sensitive data and privacy information are protected from constantly increasing threat of cyber attacks on industrial systems. The solution provides the required traceability for cybersecurity and privacy auditing to demonstrate compliance with corresponding regulations.
Implementation of digital twins with augmented reality interface may become a powerful enabler for realization of human creative potential in smart manufacturing, which is a prerequisite for transition to Industry 5.0. Digital twins are especially valuable for improving manufacturing workflow when there are decisions which are to be made by a human operator. At the same time, collection of relevant information will allow continuous process optimization which may bring multiple benefits, including better quality of products, improved energy efficiency and efficient predictive maintenance, effective integration with smart city ecosystems [24, 25].
Different levels of digitalization [26] and stages of implementation of digital twins may be supported by augmented reality assets, from a virtual copy of a separate object, which can be remotely monitored to perform quality checks and control its behavior, to a virtual twin of the entire production pipeline, which not only allows the remote control in real time, but also provides an opportunity to apply the novel methods of big data processing for the purpose of predictive analytics and process optimization. In transition to principles and practices of Industry 5.0, where human creativity will play the central role in the production processes, novel human-oriented interfaces, such as those based on augmented reality technology, will be of the utmost importance. Specific examples of digital twins, discussed in the paper, their characteristic features and possible ways of further implementation of digital twins in smart manufacturing suggest the importance of adoption and proper implementation of secure-by-design approach for digital twins design.
[11] C. Favi, M. Marconi, M. Mandolini, M. Germani. Sustainable life cycle and energy management of discrete manufacturing plants in the industry 4.0 framework. Applied Energy 312 (2022) 118671. [12] L. Bartolucci, S. Cordiner, V. Mulone, M. Santarelli, P. Lombardi, B. Arendarski. Towards Net Zero Energy Factory: A multi-objective approach to optimally size and operate industrial flexibility solutions. International Journal of Electrical Power and Energy Systems 137 (2022) 107796. DOI: 10.1016/j.ijepes.2021.107796. [13] A. Santiago, H. Fraire-Huacuja, B. Dorronsoro, J. Pecero, C. Santillán, J.J. González Barbosa, J. Soto Monterrubio. A Survey of Decomposition Methods for Multi-objective Optimization, 2014. doi:10.1007/978-3-319-05170-3_31. [14] Y. Sun, X. Li, A. Ernst, and M. N. Omidvar. Decomposition for Large-scale Optimization Problems with Overlapping Components,2019 IEEE Congress on Evolutionary Computation (CEC), Wellington, New Zealand, 2019, pp. 326-333. doi: 10.1109/CEC.2019.8790204. [15] S. Ke, F. Xiang, Z. Zhang, and Y. Zuo. An enhanced interaction framework based on VR, AR and MR in digital twin. Procedia CIRP 83 (2019) 753-758. URL: https://doi.org/10.1016/j.procir.2019.04.103. [16] Y. Yin, P. Zheng, C. Li, and L. Wang. A state-of-the-art survey on Augmented Realityassisted Digital Twin for futuristic human-centric industry transformation. Robot. Comput.Integr. Manuf. 81 (2023) 102515. URL: https://doi.org/10.1016/j.rcim.2022.102515. [17] C. Qiu, S. Zhou, Z. Liu, Q. Gao and J. Tan. Digital assembly technology based on augmented reality and digital twins: a review. Virtual Reality & Intelligent Hardware 1 (2019) 597—610. doi: 10.1016/j.vrih.2019.10.002. [18] R. Geng, M. Li, Z. Hu, Z. Han and R. Zheng. Digital Twin in smart manufacturing: remote control and virtual machining using VR and AR technologies. Struct Multidisc Optim. 65 (2022) 321. URL: https://doi.org/10.1007/s00158-022-03426-3. [19] C. Li, P. Zheng, S. Li, Y. Pang and C. K. M. Lee. AR-assisted digital twin-enabled robot collaborative manufacturing system with human-in-the-loop. Robotics and ComputerIntegrated Manufacturing 76 (2022) 102321. URL: https://doi.org/10.1016/j.rcim.2022.102321. [20] Y. Cai, Y. Wang, M. Burnett. Using augmented reality to build digital twin for reconfigurable additive manufacturing system. Journal of Manufacturing Systems 56 (2020) 598-604. URL: https://doi.org/10.1016/j.jmsy.2020.04.005. [21] R. Khan, K. McLaughlin, D. Laverty, S. Sezer. STRIDE-based Threat Modeling for CyberPhysical Systems. In 2017 IEEE PES: Innovative Smart Grid Technologies Conference Europe (ISGT-Europe): Proceedings Institute of Electrical and Electronics Engineers Inc. (2018) 1-6. URL: https://doi.org/10.1109/ISGTEurope.2017.8260283. [22] K. Wuyts, D. Van Landuyt, A. Hovsepyan, W. Joosen. Effective and efficient privacy threat modeling through domain refinements. The 33rd Annual ACM Symposium (2018) 11751178. doi: 10.1145/3167132.3167414. [23] P.K. Reddy, V.Q. Pham, B. Prabadevi, M. Liyanage. Industry 5.0: A Survey on Enabling Technologies and Potential Applications. Journal of Industrial Information Integration 26 (2021) 100257. doi: 10.1016/j.jii.2021.100257. [24] O. Duda, V. Kochan, N. Kunanets, O. Matsiuk, V. Pasichnyk, A. Sachenko, T. Pytlenko.
Data Processing in IoT for Smart City Systems. Proceedings of the 2019 10th IEEE International Conference on Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applications (2019) 96-99, doi: 10.1109/IDAACS.2019.8924262. [25] O. Duda, N. Kunanets, O. Matsiuk, V. Pasichnyk, N. Veretennikova, A. Fedonuyk, V.
Yunchy. Selection of Effective Methods of Big Data Analytical Processing in Information Systems of Smart Cities. CEUR Workshop Proceedings 2631 (2020) 68-78. [26] I. Strutynska, G. Kozbur, L. Dmytrotsa, O. Sorokivska, L. Melnyk. Influence of Digital Technology on Roadmap Development for Digital Business Transformation. 9th International Conference on Advanced Computer Information Technologies (2019) 333-337. doi: 10.1109/ACITT.2019.8780056.