In the context of the widespread application of the Internet of Things, big data, artificial intelligence, and cloud computing, intelligent manufacturing has become a development trend in the manufacturing industry. The foundation interconnection between physical space and digital space, and digital twin is the best way to achieve the fusion of physical space and digital space. This article is based on the digital twin of the intelligent factory production line, and studies the visualization monitoring technology of multi-source heterogeneous data in the production process of the assembly line. Ultimately, a virtual monitoring method for smart factories based on digital twin is proposed. To validate the feasibility of the proposed method, the Unreal Engine 4 software was used to establish a virtual monitoring system for smart factory digital twins, and the effectiveness of the method was verified, providing a reference for further realizing real-time monitoring of digital twin smart factories.
With the development of the new generation of information technology, manufacturing industry is undergoing a transformation from the physical world to the information world, realizing the interconnection and intelligent operation of the physical world and the information world, which has become the trend of the world's industrial development. An important part of intelligent manufacturing is the automated assembly line. With the widespread use of automated assembly lines, the production needs of large-scale products can be met and the economic efficiency of enterprises can be significantly improved. However, the traditional workshop monitoring methods mainly rely on manual records, 2D reports, and configuration monitoring, resulting in poor real-time and visualization. For example, manual records are error-prone, time-consuming, and do not provide real-time status information. Although two-dimensional reports and configuration monitoring have some visualization, they lack intelligent analysis and feedback
mechanisms, which limit the ability of shop floor managers.
Therefore, to address these issues, more and more companies are adopting next-generation information technology-based intelligent monitoring systems to manage automated assembly lines. These systems utilize technologies such as IoT, cloud computing, big data, artificial intelligence, and digital twins to achieve real-time monitoring, data collection and analysis, and rapid feedback and adjustment of the assembly line. Among them, digital twins feature high fidelity, multi-physics and multi-scale mapping. They establish a virtual entity that reflects the actual physical object in the virtual space, and are able to monitor the assembly line comprehensively in the virtual space with a strong sense of realism and immersion.
2023 Copyright for this paper by its authors.
For the various data generated during the operation of the assembly line, a monitoring system is
needed to store and manage the data and solve the problems such as untimely data interaction and low
visualization. Many scholars at home and abroad have conducted research and practice on this.
Although certain results have been achieved, there are generally problems such as high threshold of
system development, low development efficiency, poor system portability, and single monitoring
method, which cannot reflect the workshop manufacturing status well. For example, GuangYuan
Zhou et al [
The digital twin creates a virtual system that corresponds to the physical system. A virtual system is a complete mapping of a physical system, reflecting the operational state of the physical system. Using this virtual environment, a 3D visual monitoring system is established to provide transparent and comprehensive monitoring of the physical system. In this paper, referring to the digital twin five-dimensional model [20], the proposed digital twin smart factory virtual monitoring system is divided into five layers. As shown in Figure 1, they are physical layer, transport layer, information processing layer, virtual layer, and application layer.
The physical layer serves as the base of the entire framework, encompassing the production site environment, and comprising various devices such as robots, control cabinets, and sensors. Specifically, robots consist of robotic arms and AGVs, where the latter plays a vital role in completing the entire production efficiently by connecting different equipment to accomplish transportation tasks between different robots. On the other hand, the robot arm is the primary component that accomplishes operations by planning the end of the robot arm's trajectory to achieve tasks such as painting, assembly, welding, handling, among others. Sensory data is obtained from sensors that collect operational data from various robotic devices. Uploading this real-time data to upper-layer devices via the transport layer forms the foundation of data needed to construct the virtual layer and drive it.
The transport layer plays a critical role in enabling seamless communication between the physical and virtual layers. It serves as a bridge that allows data to flow between different platforms and systems in both directions. By transmitting operational data from sensors in the physical layer to the virtual layer through the transport layer, real-time data mapping between the two layers is achieved. Additionally, the transport layer enables the physical layer to receive commands or data from the virtual layer, thus establishing a bidirectional connection between the two layers. This layer is responsible for ensuring smooth and efficient data transmission, which is essential for creating a responsive and accurate virtual monitoring system.
The information processing layer plays a critical role in the digital twin smart factory virtual monitoring system. It processes, transforms, and fuses data collected from various sources to generate valid data that can drive the digital twin. Data processing includes data cleaning, filtering, and normalization, as well as fusing data from multiple sources to generate accurate and comprehensive data. Additionally, as the coordinate system of the robot and the coordinate system of UE4 are different, conversion at the information processing layer is required to drive the digital twin correctly. To achieve this, the information processing layer establishes a coordinate system conversion algorithm that can convert the data collected in the physical layer into data that can drive the virtual layer effectively.
The virtual layer is a crucial component of the digital twin smart factory virtual monitoring system as it provides a realistic mapping of the physical layer, reflecting its static and dynamic characteristics. It comprises a realistic model and all available data about the physical layer, synchronized with the physical layer. The virtual layer can be divided into two parts: twin data and twin model. The twin model accurately depicts the characteristics of entities in the physical layer, including location, geometry, material, color, subordination of entities, kinematic characteristics, and more. Through real-time data mapping, the twin model reflects the state of physical entities and enables monitoring of the physical layer. The twin data is sourced from the physical layer and is processed and stored as a data source to drive the twin model. Together, the twin model and twin data make up the virtual layer, providing a comprehensive and accurate representation of the physical system.
The application layer is the interface between the user and the system. The main functions of the application layer are 3D visual monitoring and operation status reproduction. The 3D visual monitoring function provides real-time monitoring and displays the operating status of the robot in 3D, including the speed, acceleration, angle, torque, and other parameters of each axis of the robot. Compared with traditional manual monitoring or video monitoring, the 3D visualization monitoring mode is more intuitive and interactive, enabling the user to monitor the scene through scene roaming in all directions without any dead angle. The operation status reproduction function enables the user to review the historical operating status of the robot and perform post-analysis, making it easier to identify any issues or areas for improvement.
UE4 is a powerful engine for simulating various virtual scenes. It is known for its powerful screen rendering capabilities and easy-to-use blueprint programming. In the digital twin smart factory virtual monitoring system,UE4 is used to create a virtual environment that is mapped to the twin data and synchronized with the physical environment. This allows the 3D model in the virtual environment to run synchronously with the real equipment and display the equipment operation information. To construct the virtual space in UE4, both static and dynamic objects need to be created. Static objects are those that do not move in the production environment, such as walls and floors. Point cloud data is generated from these static objects using radar scanning and then imported into UE4 using Point Cloud Support. The size and position of static objects are obtained from the point cloud data, and then secondary editing is performed in UE4 to create a more realistic layout. To further enhance the realism of the virtual layer scene, light sources can be added to the scene and objects can be rendered. Dynamic objects in the virtual environment include various types of robotic arms, AGVs, and conveyors. For AGVs and conveyors, there is no complex hierarchy, so only the geometric model, physical model, material, and color are modeled in 3Dmax software and then imported into UE4 using the Datasmith plug-in. Robotic arms, on the other hand, have a complex hierarchy of components, including the base, swivel, large arm, small arm, small arm bar, wrist, and end flange. The kinematic arm hierarchy is described in UE4 so that the virtual model corresponds to the physical entity characteristics. (b) Figure 2: Virtual environment construction: (a)Physical space.; ( b)Virtual Space
Robot end trajectory visualization is the process of presenting the robot end trajectory graphically. The end trajectory visualization provides an intuitive understanding of the robot's working state and motion trajectory, which helps to analyze and optimize the robot's motion planning and control. In UE4, a particle system can be used to present the visualization of the robot end trajectory. First, create the particle system, set the shape, size and color of the particles, and then, attach the particles to the end-effector of the robot through the Spawn Emitter Attached function in order to make the particle effect move along the trajectory of the robot's end, as shown in Figure 3.
The physical space and virtual space are on ROS and UE4 platforms respectively, so the barrier between UE4 and ROS needs to be broken. And RosBridge of ROS platform provides JSON interface between ROS and non-ROS platforms to realize the data communication function between ROS and UE4 platforms through various communication methods. ROSIntegration is used in UE4 to connect UE4 and ROS. it uses itself as a client in ROS system and uses ROSBridge as a communication intermediary to connect to ROSBridge server with WebSocket protocol to realize two-way communication between ROS and UE4, and its communication framework is shown in Figure 4. Specifically, ROSIntegration maps ROS topics into the data structure of UE4, enabling users to directly subscribe to ROS topics and process the corresponding data in UE4. It also supports calling ROS services in UE4, and users can create a ROS service client object and specify the service name and service type to call the corresponding services in the ROS system. Overall, ROSIntegration acts as a communication bridge to connect ROS and UE4 organically, allowing users to use ROS functions in UE4 and to control the robot and process data in the ROS system in a more flexible way.
Twin data is a critical component of digital twins, which includes both the static and dynamic data that build the physical space. Static data encompasses information such as the geometric shape, size, and layout of the device, usually transformed from real-world data captured by sensors. Dynamic data, on the other hand, is the operational data generated during device operation and is a key part of achieving real-time mapping. In this article, the dynamic data mainly comes from the UR5 robot and AGV car.
The twin data of UR5 robot includes current timestamp, joint name, joint position, joint velocity, joint acceleration. The message format of its dynamic data is shown as follows: std_msgs/Header header unit32 seq time stamp string frame_id string[] name float64[] position float64[] velocity float64[] effort std_msgs/Header header unit32 seq time stamp string frame_id float64[] x float64[] y float64[] theta
The dynamic digital twin data of AGV includes timestamp, position, and direction in space, with the specific format as follows:
After transferring data between ROS and UE4, coordinate transformation is required because the coordinate systems used by the two platforms are different. In UE4's left-handed coordinate system, the thumb points towards the positive X-axis, the index finger towards the positive Y-axis, and the middle finger towards the positive Z-axis. However, in ROS's right-handed coordinate system, the thumb points towards the positive direction of the X-axis, the index finger towards the positive direction of the Y-axis, and the middle finger towards the positive direction of the Z-axis. Figure 5 shows the coordinate systems used by UE4 and ROS.
Because the two platforms, UE4 and the robot, are in different coordinate systems, conversion is necessary for calculations. First, the point in the left-handed coordinate system is transformed into the right-handed coordinate system. Second, rotation is performed based on the rotation matrix in the right-handed coordinate system. Finally, the rotated point is transformed back into the left-handed coordinate system. As shown in Figure 5, the X-axes of the two coordinate systems point in opposite directions. Therefore, the point PR ( x, y, z) in the right-handed coordinate system, represented as PL ( x, y, z)
in UE4, can be expressed as a matrix: The rotation matrix for the right-handed coordinate system is expressed as follows: x 1 PL y ST PR 0 z 0 0 0 x 1 0 y 0 1 z r00 RR r10 r 20 The rotation matrix converts the points PR ( x, y, z) to PR (x, y, z) : (b) Figure 5: Coordinates system.: (a)Coordinates in UE4 space.; (b)Coordinates in physical robot space.
The data collected and transmitted from the ROS-side robot can be categorized into two types: real-time data and historical data. Real-time data is mapped in real-time based on the physical and virtual layers, driving the virtual model to run for comprehensive monitoring and management of the equipment operation site. Storing historical data provides users with a data query function. Users can query the operation status of the equipment within a certain period of time and reproduce the working process of the equipment in the virtual environment. The stored data can be used for post-event analysis, further laying the foundation for future analysis and optimization, and intelligent decision-making using big data. The data serves as the basis for data traceability and error identification when the equipment operates abnormally. The reproduced data includes the robot's joint angle, joint speed, joint acceleration, and joint torque.The details of the fields in the database table are shown in Table 1.
INT DOUBLE DOUBLE DOUBLE DOUBLE
number jointAngles jointVelocity jointAcceleration jointTorque
This system development environment is divided into physical space development environment and virtual space development environment. The virtual space is developed based on Windows 10, and MySQL is used as the database software to build the database, and UE4 software is used for the 3D virtual simulation, and SW (SolidWorks) and 3Dsmax software are used for 3D modeling. The physical space is developed on Ubuntu 16.04 linux system, and Robot Operating System (ROS) is used to control the robot and collect its operation data [21]. The specific development environment is shown in Table 2 and Table 3.
The experimental procedure is to send a motion command from the controller to the robot drive, which drives the robot's motion. The entire operation of the robot is monitored in the monitoring system.
The specific steps of the experiment are as follows： 1. Start the rosbrige_server in Rosbridge to open a TCP server. 2. Click on the Run button in UE4 to connect the monitoring system to the socket server on the ROS side and start the monitoring process. 3. The controller receives the command and then converts it into a drive signal to the driver, which drives the robot's internal motors. movement and the physical robot begins to move. 4. The controller transmits the robot's real-time operational data to the monitoring system via socket communication; it checks the consistency and synchronisation of the model's movement with the actual robot in the monitoring system.
The paragraph describes the results of experiments conducted to evaluate the proposed system's performance in 3D visualization and monitoring of a robot, as presented in Figure 6. The virtual model successfully tracks the robot's movements and displays the real-time operational status of the device entity. The system's interactive features allow users to roam the monitoring scene, adjust the viewpoint distance and angle, and switch the perspective as desired. The experiments also found that the system has no noticeable delay and provides good real-time performance, enhancing its ability to monitor scenes in 3D. Overall, the results demonstrate that the proposed system effectively enables 3D visual monitoring of a robot in real-time, providing a useful tool for users. (b) Figure 6: Monitoring process: (a)Physical space.; ( b)Virtual Space
To demonstrate the real-time capability of the digital twin system described in this article, we have collected the timestamps of four adjacent data points in each of the three stages, with the system runtime as the reference zero point. These timestamps are listed in Table 4.
Based on Table 4, it can be observed that the average time intervals between adjacent data collections for stages A, B, and C are 0.207s, 0.204s, and 0.167s, respectively. This further confirms the excellent real-time performance of the proposed digital twin system in this paper.
The paragraph explains the process and results of run state reproduction experiments. During these experiments, the robot's real-time data was recorded while it was in operation. This data was later called from the system's database to reproduce the robot's state and verify whether the reproduced state was consistent with the actual running state. To ensure data accuracy, the historical data was compared with the real-time data sent by the controller. For instance, Table 5 shows an example of historical data fields, J1Angles to J6Angles, representing the received angle values in degrees for each of the six axes, while Table 6 displays the real-time data sent by the controller.
The above data was inputted into the formula for mean absolute percentage error, and the calculated error between the replicated state and the actual operating state was only 0.1465%, demonstrating the reliability and accuracy of the data.
SR 100% n n i1
VJi P
PJi
Ji 0.1465%
R denotes the average absolute percentage error between the historical data and the actual run data. VJi (i= 1,2,3,4,5,6) is the joint angle of the historical data PJi (i= 1,2,3,4,5,6) is the joint angle of the actual run data.
J2Angles where S
J6Angles -120.091° -120.091° -120.091° -120.091° -120.091° -120.091°
J6Angles -120.079° -120.079° -120.079° -120.079° -120.079° -120.079° (7)
This paper describes a novel approach to building a smart factory virtual monitoring system using digital twin technology, which offers several scientific and practical innovations. The proposed system architecture consists of five dimensions, with detailed explanations of the digital twin system components. The use of UE4 to construct the virtual scene provides a more realistic environment than previous virtual monitoring systems, enhancing the accuracy and precision of monitoring operations. The system facilitates real-time data mapping between the physical and virtual space, enabling the monitoring of industrial robot assembly line operations in the virtual environment, which is a significant advancement in digital twin technology. The use of MySQL for storing historical data allows for the reproduction of historical motion states, enabling users to analyze past performance and optimize future processes. Overall, the proposed system offers an innovative solution for monitoring and managing smart factories, improving efficiency, and enhancing overall production processes. The system's advanced features, such as real-time data mapping, digital twinning, and historical data reproduction, provide users with valuable insights into the factory's operations, enabling them to make data-driven decisions and optimize processes, thereby enhancing the practical value of the system.
[14] S. Aheleroff, X. Xu, R. Y. Zhong, Y. Lu, Digital twin as a service (DTaaS) in industry 4.0: an architecture reference model, Advanced Engineering Informatics 47 (2021) 101225. https://doi.org/10.1016/j.aei.2020.101225 [15] A. F. de Souza, J. Martins, H. Maiochi, A. D. P. Juliani, P. A. Jaskowiak, Development of a mobile application for monitoring and controlling a CNC machine using Industry 4.0 concepts. The International Journal of Advanced Manufacturing Technology 111 (2020) 2545-2552. doi:10.1007/s00170-020-06245-2 [16] A. H. Sabry, F. H. Nordin, A. H. Sabry, M. Z. A. Ab Kadir, Fault detection and diagnosis of industrial robot based on power consumption modeling, IEEE Transactions on Industrial Electronics, 2020, 67(9):7929-7940. doi: 10.1109/TIE.2019.2931511 [17] J. C. Xie, X. W. Wang, S. Q. Hao, et al, Operating system and key technologies of transparent fully mechanized mining face driven by industrial Internet, Computer Integrated Manufacturing Systems 25 ((2019) 3160-3169. [18] L. Liu, H. X. Du, H. F. Wang, T. Y. Liu, Construction and application of digital twin system for workshop production process, Computer Integrated Manufacturing Systems 25 (2019) 1536-1545. [19] H. Zhao, J. H. Liu, H. Xiong, et al. Real-time monitoring method for three-dimensional visualization of digital twin workshop. Computer Integrated Manufacturing Systems, 25 (2019) 1432-1443. [20] F. Tao, W. Liu, M. Zhang, T. L. Hu, et al. Five-dimension digital twin model and its ten applications, Computer integrated manufacturing systems 25 (2019) 1-18. doi: 10.13196/j.cims.2019.01.001 [21] Z. Wang, Y. Ouyang, O. Kochan, Bidirectional linkage robot digital twin system based on ROS, in: Proceedings of 2023 IEEE 17th International Conference on the Experience of Designing and Application of CAD Systems, CADSM 2023, pp. 1-5. doi: 10.1109/CADSM58174.2023.10076497.