This study is devoted to the problem of digital object detection of construction site based on Building Information Modeling (BIM) technology and Artificial Intelligence (AI) systems. To detect, classify and evaluate individual components and objects on a construction site, an artificial intelligence system is proposed that combines Convolutional Neural Network (CNN) and Feed-Forward Neural Network (FFNN) architectures. The authors propose the using of the Internet of Things (IoT) and Big Data technologies for real-time recognition of building objects on a construction site. The effectiveness of identifying a set of data attributes from a BIM model to create a digital twin of a construction site has been confirmed. The reliability of object recognition by the CNN model was proved with an indicator of 90.4%, which confirms the correlation of the FFNN model with the attributes of the reference BIM model. The results of this study can be used to further improve the concept of creating digital twins of construction objects throughout the entire life cycle and to monitor real estate objects at the operational stage.
Nowadays, the integration of Building Information Modeling (BIM) technology in the construction industry is used for a huge range of operations and life cycle processes, including environmental monitoring; management of facilities and their processes; energy efficiency assessment; control of construction processes, etc. The rapid development of a complex of information technologies, in particular artificial intelligence systems, IoT, and Big Data technologies, allows us to assess the enormous prospects for the development of BIM technologies, including in the context of the introduction digital twin of the construction space.
This research consists of the scientific substantiation of the combination of a complex of information technologies, in particular an artificial intelligence system, which at different stages uses two conceptually different architectures, namely: CNN, which uses the YOLOv3 model to detect a set of objects in the image in real-time, and FFNN, which performs multi-label classification according to the received and reference indicators of BIM design. Additional, but no less important, within the framework of the study, are the aspects of the inclusion of IoT and Big Data technologies in the system, which comprehensively allow obtaining enormous arrays of useful data, which, in combination with the mechanisms of the artificial intelligence system, can create a plausible digital twin of the construction site. The developed digital twin can be used to systematize and assess the quality of all construction life cycles.
The theoretical analysis of the evolution of BIM technologies and the development of complex
information technologies, which contributed to the creation of digital twins is engaged by the team of
scientists at Michigan University in the composition by Min Deng, Carol Menassa, and Vinet Kamat.
In their work [
In scientific papers [
A group of Chinese scientists consisting of Hao Wu, Lingbo Liu, Yang Yu, Zhenghong Peng,
Hongzan Jiao, and Qiang Niu is engaged in the research on the effectiveness of using Big Data
technology in urban studies. The paper [
In the article [
The duo of scientists Liquan Zhao and Shuaiyang Li are engaged in research aimed at improving
object detection algorithms, in particular with the help of the YOLOv3 model. In the article [
In the publication [
The research on the process of the multi-label classification is carried out by a team of Ukrainian
scientists in the composition of S. Dolhopolov, T. Honcharenko, S. A. Dolhopolova, O. Riabchun, M.
Delembovsky, and O. Omelianenko. In the paper [
This study aims to develop an information system that integrates BIM technologies, CNN, and FFNN systems of artificial intelligence, IoT, and Big Data technologies, which will allow the creation of a digital twin of the construction site. To achieve the goal of the study, it is necessary to characterize the aspects of BIM design that will allow the real-time collection of data by IoT and Big Data for intelligent video surveillance using the YOLOv3 method and multi-label classification of attributes BIM of the model of the future digital twin.
Having analyzed the works of Ukrainian and foreign scientists, we have developed a model illustrating the natural way to achieve a high-quality digital twin. Fig. 1 shows the developed “Key to Digital Twin” mod.el
According to the developed “Key to Digital Twin” model, data in “raw” form is collected on construction site using the IoT complex, which in turn is supplemented by Big Data and implements the flow of raw, but most valuable, information. On the other hand, in asynchronous mode, specialists from different fields form a BIM “Reference Model” of the construction object, the attributes of wh are explicitly or implicitly entered into the BIM Data Storage and remain in standby mode until YOLOv3 performs the classification of actual data on the construction site. After both parts are ready, the FFNN model is used to match attributes, i.e. multi-label classification of matches. As a result, artificial intelligence provides a percentage correspondence of the actual results of construction processes and the state of the construction site in comparison with the reference object designed with the help of BIM technologies by multiple parameters. According to these data, a decision is made to create a digital twin using CAD systems using the defined data. In the future, the developed system can support the digital twin by processing information in real-time, which will provide an opportunity to make operational adjustments.
The study uses YOLOv3, which is a modified version of the general YOLO architecture, consisting
of 106 convolutional layers and capable of recognizing small enough objects. The main advantage of
YOLO 3 is that it has 3 output layers, each of which filters objects of different sizes [
To implement the search of some array of BIM objects of the construction site, we will form a small test Dataset consisting of 5 different classes, where the total number of images for each class is 100 units. Thus, the trained neural network model will be able to distinguish between 5 different objects at the construction site. Namely: BIM1 is iron reinforcement; BIM2 is worker (man); BIM3 is protective helmet; BIM4 is vegetation; BIM5 is earth embankments. In actual BIM design, there are many more attributes that can be recognized and classified with YOLOv3. The YOLOv3 model is trained in the ratio of 80% of training images, and 20% of test images.
List of libraries and software tools that were used to test the proposed approach in the research: python >= 3.6, numpy, torch >= 1.0.0, opencv, json, keras, pandas, numpy matplotlib, tensorboardX, CUDA.
It is necessary to determine the average precision per object and the average precision of the entire model to further verify the correctness of training the YOLOv3 model. The value of the average precision (AP) object is determined by equation: (1) (2) where AP is Average Precision; is the i-object classification value; is the definition of the time series of the corresponding class; ( ) is the derivative interpolation function that determines the largest value in the time series.
The value of the average precision of the entire model (mAP) is determined by equation: where APi is Average Precision of i-class; is mean Average Precision; is the number of defined classes to recognize.
Note: (1) is necessary to derive (2), but in the general case, it is a time series of the corresponding class that is recognized, among all classes available in the model, in this case, it is: BIM1, BIM2, BIM3, BIM4, BIM5. Thus, the model selects the maximum value from the time series, which allows us to conclude that the object is the most consistent with a definite class for each box (of each cell) and extract a probability that the box contains a specific class.
After completing the training of the YOLOv3 model, we use part of the test material in manual mode to verify the correctness of the neural network. Fig. 2 presents one of the test materials that demonstrates the proper operation of the model by the identification of 3 available classes, namely: BIM1 – iron reinforcement; BIM2 – worker (man); BIM3 – protective helmet.
The next step is to compare the obtained indicators with the reference ones using the FFNN model, which receives and processes the input data using the RELU activation function and provides the output data using the Soft plus activation function [15]. Thus, the multi-label classification for reference and actual indicators on the construction site is realized.
∑(
) ( )
( ( )
) ∑
For the operation of the FFNN model, pre-defined reference indicators according to the specified classes using the method of analysis of hierarchies, namely: BIM1 – 70%, BIM2 – 100%, BIM3 – 100%, BIM4 – 91%, BIM – 84%.
In the last stage, with the help of (1) and (2), the mAP of the YOLOv3 model and the FFNN model, and the overall mAP indicator of the system are determined.
The training and validation results for the specified set of classes are shown in Fig. 3. Training continued for 25 generations, where one could observe a tendency to reduce both training and testing errors.
Fig. 4 shows the graphical distribution of the average precision of classes according to the results of model YOLOv3, which are: BIM1 – 81%, BIM2 – 96%, BIM3 – 97%, BIM4 – 90%, and BIM5 – 88%. Thus, according to (2), the mean average precision of the model is:
Thus, according to the results of the study, the average precision of the FFNN model for each classified YOLOv3 class was determined. Table 1 represents a generalized set of all initial data used in calculating the mean average precision of the system, as well as evaluating its suitability for approval in the project of a digital twin.
Thus, the Etalon BIM was set for each class manually as a definite limit of accuracy was required within the project. AP FFNN demonstrates the calculation of compliance of the obtained accuracy results with Etalon BIM.
Thus, the mean average precision of the YOLOv3 model (AP YOLOv3) concerning all recognized classes is 90.4%, and the mean average precision of the FFNN model according to the ratio of reference and actual data is 98.38%. Thus, according to the obtained results, the mean average precision of the FFNN demonstrates that the mean average precision of the YOLOv3 exceeds the mean Etalon BIM by 10.54% and demonstrates that the specified objects of the construction site are successfully identified and can be represented in a digital twin.
According to the study carried out, we consider it appropriate to recommend the developed information system that combines artificial intelligence systems, Big Data, IoT, and BIM technologies and allows for systematizing and evaluate an enormous set of parameters used in the design of BIM and other stages of the life cycle of the construction space.
As part of the study, the “Key to Digital Twin” model is developed and presented, which allo better systematize and understand the evolutionary path of BIM, and, at the same time, includes a set of combinations of two artificial intelligence systems, namely CNN and FFNN, which allow to recognize arrays of objects in real-time and compare them using multi-label classification. The developed software includes the ability to integrate all the latest information technologies, namely: IoT and Big Data, which allow to accumulation and process of an enormous amount of data related to the construction site. According to the results of the YOLOv3 artificial intelligence system, its effectiveness in recognizing objects that are pre-characterized as BIM classes during design is determined. Thus, the mean average precision (mAP) of this system is 90.4%. At the same time, the artificial intelligence system based on the FFNN model provided more accurate indicators by the BIM benchmarks, namely 98.38%. Therefore, according to the obtained results of mean average precision FFNN demonstrates that the mean average precision of the YOLOv3 exceeds the mean Etalon BIM by 10.54%, which allows us to conclude that the information system works properly and can provide correct data for further coordination and design of digital twins.
“Digit
through dee
[13] R. V. Sevilla, A. S. Alon, M. P. Melegrito, R. C. Reyes, B. M. Bastes, and R. P. Cimagala, “Mask-Vision: A Machine Vision-Based Inference System of Face Mask Detection for Monitoring Health Protocol Safety,” 2021 IEEE International Conference on Artificial Intelligence in Engineering and Technology (IICAIET), pp. 1-5, October 2021. https://doi.org/10.1109/IICAIET51634.2021.9573664 [14] T. Honcharenko, V. Mihaylenko, Y. Borodavka, E. Dolya, and V. Savenko, “Information tools for project management of the building territory at the stage of urban planning,” CEU Workshop Proceedings, 2851, pp. 22-33, 2021. URL: https://cutt.ly/bI3fAEo [15] P. Szymański, and T. Kajdanowicz, “-Smcui kltiitlearn: a scikit-based Python environment for performing multi-label classification,” The Journal of Machine Learning Research, vol. 20, no. 6, pp. 209-230, December 2019. URL: https://cutt.ly/vCOGhw