-Due to the growing need for electricity, the effective inspection of the power lines is becoming an important matter. In this paper, The author presents the inspection of power or transmission line with autonomous automatic UAVs (Unmanned Aerial Vehicles). For the comprehensive inspection of power lines and its different components, such as (cross arms, cracks in poles, rot damage, and insulator burn), It is needed to inspect from every side of the elements and the masts. So, the angle and speed of the drone are much more important to take images while moving around the poles. The simulator used for the experiments, including with the deep learning models,which acts as a vital source of data analysis. At the same time, the pictures used as the primary data source. Through the Deep learning method, a suggestion of action generated for the movement around the masts. The use of a simulator is a quick, accurate, and inexpensive solution, with less real/world factors affecting the inspection process, such as weather, time, and cost of using a large number of different resources. This study presents experiments with lightweight deep learning models through developing the prototype of vision based unmanned aerial vehicle to inspect the power line in a simulated environment. It focuses the large demand of power companies to inspect the power line autonomously with the influence of deep learning. Finally, Several deep learning models are compared when inspection along the power lines. The model shows satisfactory results in the testing path. The model trained by the MobileNetV2 performs best among all other models. Index Terms-Power line Inspection, Vision-based model, Deep learning, Unmanned Aerial Vehicle (UAV), Drone
Due to the increase of research effort in the field of aerial
robotics, the application scenarios of the Unmanned Aerial
Vehicles(UAVs) are growing very fast during the last few
years[
Manual traditional patrol is the most extensively used inspection method even it is tedious, long and exhausting, a team inspects the transmission lines from the ground with the help of different instruments such as binoculars, infrared cameras.
It is an expensive but fast method for inspection of transmission lines, the pilot flies the helicopter, and the camera operator takes the recording of the power lines with color, infrared and thermal cameras and its various components like insulators, masts, and conductors. After that, the fetched data are manually inspected by the workers with the help of a machine.
Previously the focus is on the low-level local features of
gradients, texture, and brightness. It separates potential pixels
of lines from the background by using the Canny and Sobel
edge detector and then applied the [
To reduce the cost and the risk of life, another solution is
advised, which is climbing robots. It is faster, less expensive,
and safer than the foot patrol and travels along the conductors,
but having these huge benefits it is not a practical solution to
inspect the vast network of lines[
The drone simulator or simulated environment is used to run the experiments because it takes less time as compared to the real environment to collect the training data and to minimize the costs to establish the experiments. Given the increasing importance of the topic, in this paper author presents the data acquisition of the power line inspection using UAV’s.
Our approach used deep learning models in the acquisition of data for the inspection of power lines, providing it with new understandings on the topic.
The remainder of this paper is ordered as follows: In section 2, background and related work. In section 3 method and result is discussed. Section 4 presents the conclusion.
A development in the field of deep learning CNN based model[21] has proved an astonishing performance [36]. These methods demonstrated the ability to learn multiscale features.
R.Madaan suggested the framework using a dilated
convolutional network[22] and treated the lines as the semantic
segmentation task. Several networks are designed using the
dilated convolution with various architecture and evaluated
to find the optimal one. The performance improves a lot
compared to traditional methods and efficient on NVIDIA
Jetson TX2[
II. BACKGROUND AND RELATED WORK should be conducted by the UAV having different sensors and Power lines are inspected traditionally by the foot patrol and cameras to detect the faults, and the same method and data by helicopter after regular intervals. A team is carried out the can be used for the offline inspections. So, the outcome of inspection on foot or with the help of a helicopter to collect the inspection methods suggests different possibilities for the data for further visual inspection for the power lines. Enhance most frequent issues, such as high cost, speed and safety. accuracy and speed, a significant amount of research has been As compared to the other inspection methods which are carried out to systematize vision-based power line inspection. mentioned above, UAV inspection cost is relatively low and it can fly close to the power lines to take the comprehensive A. UAVs Supported Method views/images of the different components which can improve
The advanced and fast way to inspect the power lines the accuracy. V.N.Nguyen also mentioned the challenges of
from the required distance is the UAV supported method. power line inspection using automatic and autonomous UAVs,
According to recent development in-flight handling techniques, together with the possibilities and disadvantages[24].
UAVs now equipped with proper payloads (thermal and visible According to [25] there are three approaches regarding the
cameras)[
It is controlled by varying the speed of four rotors independently, the pitch movement is obtained by changing the ratio of back and front rotor speed and a roll by altering the right and left rotor speed. The yaw movement is achieved by torque resulting from anti-clockwise and clock-wise speed[30]. The roll and pitch movement are obtained due to the difference between front and back rotors. The equation is written below:
The ( 2) is the second actuator output and ( 3) is the third actuator where t is the thrust from the ith rotor and l m is the distance from the centre of mass.
2 = l( 4 3 = l( 1 3)(N m) 2)(N m)
The first and the final actuator can be defined by the equations below: 1 =
1 + 2 + 3 + 4=m (m=s2) 4 = 3 + 4 1 2 (N m) they are not suitable for high-speed vision-based navigation, pole based navigation attracts less attention, and this is due to the lack of information for navigating between the poles. Towards the vision-based automatic, autonomous inspection, deep learning is the data analysis approach because of the following points[24].
• In Deep learning algorithms, Convolutional Neural Network improved to a great extent, and the performance of visual recognition systems for many applications such as self-driving cars, image search is remarkable.
• It provides automatic learning features that can reduce the effort in solutions for every subtask in power line inspection.
• The ability of the generalization in deep learning opens the possibilities for vision-based inspection, such as the model trained for a specific task that can be used in the related tasks.
VS, also known as the vision-based remote control, is a technique that uses response information coming from the different visual / vision sensors to control the movement of the UAVs. Some line based visual servoing is of a UAV is presented in the [27]. Araar proposed two solutions to the classical IBVS formulation, which is improved by using LQServo control design and the partial pose based visual servoing,which they get from pure visual measurements.
The inspection of power lines can be divided into two modes: tower monitoring and line monitoring. In tower monitoring, we monitor the breakage of all kinds of clips, defects deformation, shock hammer damage, So the task requires the precise positioning of UAVs and the stable flight to gather the high-quality image information. Line monitoring includes the inspections of the trees and the breakage of line, tilted, and collapsed tower[28]. There is a strong electromagnetic field near the power line; it may damage the UAV electronic equipment’s so the minimum safety distance is required for UAV inspection, and for the safety consideration J.Cui [28] proposed the horizontal and vertical range. Based on the regulation and experience, the safe detection distance s=8 m and the length=10 m.
R1 = Ltan( =2) = 10tan((21:28=180)( =2)) = 1:87m (Horizontal direction)
R2 = Ltan( =2) = 10tan((15:92=180)( =2)) = 1:39m (Vertical direction)
X. Qin proposed a method of detecting inspection objects through LiDAR data, which have four steps. First, the point cloud is divided into the single-span as the processing unit.Secondly threshold is created to remove ground points which improves the data extraction efficiency. In third step, surrounding data of the line can be extracted by the position and orientation system, and finally, the partition recognition (3) (4) (5) (6) (1) (2)
Every rotor in the UAV produces a force to lift the UAV and its moment, the (1,3) rotor and (2,4) rotor rotate in the opposite position to cancel the effects generate by the other pair. To make a roll angle , increase the angular velocity of rotor 2 and decrease the 4 while keep the thrust constant, in the same way one can get the pitch angle by increasing the 3 and decreasing the 1 rotor and yawing angle is produced by increasing the speed of (1,3) and decrease the speed of (2,4) the mechanism is shown below[31] in Fig 2.
Deep learning attracts the attention of the power line
companies because of vision-based power line inspection. It also
covers an extensive range of faults on a single inspection,
vision-based inspection systems explain in [32], and [24]. The
authors proposed the images as the potential data source for
vision-based inspection because it provides sufficient
information for detecting the wide range of common faults on the
components of power lines, it’s easy to collect and easy to
analyze[
With the current advances in deep learning, UAV technology is accurate, reliable, fast, and safe for power line inspection. Our system uses a vision-based power line inspection method with the UAV as the leading method and images as primary data sources and deep learning for the data analysis.
Micro aerial vehicles are mostly used GPS for their navigation in recent years, and they have very little ability to avoid the obstacles. Bachrach at al. [33] using an RGBD camera to build a map for planning and localization. Bry et al. [34] used the Inertial Measurement Unit (IMU) with laser a finder in an enclosed environment for a reliable flight. For state estimation and map building, Fraundorfer et al.[35] used a downward-facing camera with forward-facing stereo pair cameras. Scaramuzza et al.[35] used three cameras with Inertial Measurement Unit (IMU). The primary responsibility of the drone is to follow the power lines; the auto-drone developed in a way to handle each image input at a time and calculate the waypoint to move the drone in a particular position [35]. Our research is conducted in a drone simulator, which has the power line environment in it. Our system handles the images input continuously and then produces the commands to move the drone. As mentioned earlier, the images are the primary source/input, and the steering commands are the output of the system. In all these scenarios, the deep learning models have significant importance as it needs for getting good accuracy, and this estimation passed to a simple controller to fly the drone in the right way. Fig 3 demonstrates the mechanism.
In our system, author used UAVs as the primary method for the data acquisition of the masts(pole) in a simulated environment. The general idea is to take the images of every mast (pole) from a single camera.
The drone used to collect the data. It moves above the pole and waits to hover and move forward. The collected data was not good enough because it needs data from every side and by using drone hovering. Still, it only get the data/information that is only in the direction of the camera, as shown in the Figure 4. Our drone camera permanently faced with only one direction; it is not a revolving camera. The data collected was not useful for us because it is necessary to take the images from every side of the masts. So, the author moved forward to another implementation.
After the first scenario hovering, author took another step to move the drone in a triangular manner, as shown in the Figure 5. It shows some good results. It was not enough for us because some sides of the masts are still missing, and it has the high risk of losing some portions of the masts. If some of the sides are still missing, it will affect our results and requirements, but we tried our best to decrease the errors. After gathering this information, author changes the approach toward data gathering, while putting these approaches behind and move to the new way to collect the data. The primary source of input is images, and now we want to move the drone on all sides of the masts and take images from every side. So, considering the requirements, we move to the next step.
After investigating hovering and the triangular path, rectangular path is considered to take next step. The figure 4 demonstrate the rectangular movement of the drone to collect the data. Drone move from one corner to another to collect data. This strategy is adequate than the previous two, but it has still some limitations in it. The thing is that the drone moves from one corner to other and then adjust its position in front of the masts and take pictures. The reason is that the camera, which is mounted on the drone, only facing the front side it is not movable camera like most of the drone. So, on every corner it must face directly to masts to take images. While moving it only looks on the front side and the mast is not in the frame to take the pictures for data collection. It takes too much time to move from one position to another and then adjust the camera to take picture. After examining the pictures, author come on the conclusion that it is better than the other two approaches. But still missing some angles to take the pictures. drone and then split in training and validation set.
In the data acquisition model, the drone is at some position n and moves all around the poles.
The first step is to know the exact position of the drone and then move one step in a clockwise or counterclockwise direction with a constant radius.
The next point is y, in fig 6, and it moves from n (if the drone is on n) to y, then the angle between (n, y) represented as delta . And at point ‘y,’ the drone repeats its function with the new coordinates and completes the circle around the masts(pole) to collect the images.
After that, the drone moves all around the masts to collect the images. These images are the combination of different locations/angles of the camera and a drone. After experimenting with the several executions of the drone controller, author reached the implementation, which is quite stable. The maximum velocity of the drone is determined at 1 m/s. The drone is used to collect the training data, which is automatically controlled by the script with the AirSim APIs. The drone is designed to keep a constant distance to the masts. Data collection is conducted on every mast(pole), the angle of the camera is 45 degrees, and the height of the camera is fixed, which is 5 points above the masts(pole). At every 5 degrees movement/change, the drone takes the image it moves all around the masts(pole), as shown in Figure 7.
Deep learning models needs a large amount of data to train well, so; there are about 44,286 images collected from the The flow of the controller is represented in the fig 9. After the inspection model it produces some output and these outputs are handled by the controller to move the drone in a specific direction. Based on the Estimation position and angle the goal of the controller to maintain the drone in a right position.
The author have a lot of images data, which is expensive to load directly into the memory, so DataGenerator is used. This concept of DataGenerator; it’s like an iterator, which reads data in chunks from the disk. The idea of (ROI) region of interest is used when generating the batches, remove the image piece, which is not essential for us. The model adds or removes the brightness from the images and randomly flip the images, so the model has some new information to learn. Here you can see them in the image 10 with the rectangle in which we are interested in.
After training the models, the sanity check is performed and load some of the images to compare the Actual and predicted angle and got some values, as shown in the following figures.
The actual angle in fig 9 is 45 degrees, but the predicted angle after training is 44.85 degrees, and the error is 0.142, and in fig 8, the actual angle is 15 degrees. Still, the expected angle is 14.25, and the error is 0.74, and the third fig 10 shows some more error rate, which is 1.64, and the actual angle is 21 degrees, and the predicted angle is 19.35. Author applied some different deep learning models on the dataset like MobilenetV2, ResNet50, DenseNet121, NASNetMobile to compare the inspection of the power lines, so some of the training graphs of the model are shown below:
There is a decrease in the plot of loss function of MobilenetV2 graph which is shown in figure 11 and the plot of validation loss decreases and has a gap with loss, and it looks quite stable from start to end, but on the other hand, the graph of DenseNet121 is little change at the end of epoch, the loss rise at this point. The tests are conducted for the automatic data acquisition of the power line inspection model. There are four inspection model are involved in the test MobilenetV2, ResNet50, DenseNet121, NASNetMobile are shown in table.
effort is increasing in the field of aerial robotics. But power line companies are interested in multirotor UAVs. Our modernday societies are dependent on electricity so, the inspection and monitoring of the power lines are extremely important. The main motivation behind this work was to automatic acquisition of data using autonomous UAVs on the power line inspection and through all these things in mind author selects simulated environment for this purpose.This paper presents the power line inspection system that uses UAVs for inspection, deep leaning as the data analysis, and images as the primary data source. There is not much difference in the training result of the different models. But the results of the tests show the difference in the performance of the inspection model. It’s found that MobileNetV2 stands first in the test, with three times interventions among four inspection models tests. It is lightweight model between other inspection models and performs good in all four other models. It needs least amount of intervention in autonomy tests. The grading of the other inspection model as this order: DenseNet121, ResNet50 and NasNetMobile. It is seemed that the performance of the models for the inspection is satisfactory. The research shows that our inspection model assists the tests of the power lines and provides some better results without any more adjustment methods such as sensors and camera adjustment.
To enhance the power line inspection by the autonomous UAVs, first need the object detection model, then it can make progress in developing the automatic capturing function. The primary task of the function is to capture and detect the faults and do it automatically. After doing all this research, the next step is to apply this autonomous UAV power line approach in a real power line environment. The data collection method in a real environment could be a challenge due to the gap between the actual and simulated environment. The inspection model trained on the synthetic training images used in the real environment. However, it is in the simulated environment; we want to extend this work in the real environment for the power line inspection method.
[21] Wei Shen, Xinggang Wang, Yan Wang, Xiang Bai, and Z. Zhang, “DeepContour: A deep convolutional feature learned by positive-sharing loss for contour detection,” in 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA, 2015, pp. 3982–3991, doi: 10.1109/CVPR.2015.7299024. [22] R. Madaan, D. Maturana, and S. Scherer, “Wire detection using synthetic data and dilated convolutional networks for unmanned aerial vehicles,” in 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Vancouver, BC, 2017, pp. 3487–3494, doi: 10.1109/IROS.2017.8206190. [23] J. I. Larrauri, G. Sorrosal, and M. Gonzalez, “Automatic system for overhead power line inspection using an Unmanned Aerial Vehicle x2014; RELIFO project,” in 2013 International Conference on Unmanned Aircraft Systems (ICUAS), Atlanta, GA, USA, 2013, pp. 244–252, doi: 10.1109/ICUAS.2013.6564696. [24] V. N. Nguyen, R. Jenssen, and D. Roverso, “Automatic autonomous vision-based power line inspection: A review of current status and the potential role of deep learning,” Int. J. Electr. Power Energy Syst., vol. 99, pp. 107–120, Jul. 2018, doi: 10.1016/j.ijepes.2017.12.016. [25] G. Zhou, J. Yuan, I.-L. Yen, and F. Bastani, “Robust real-time UAV based power line detection and tracking,” in 2016 IEEE International Conference on Image Processing (ICIP), Phoenix, AZ, USA, 2016, pp. 744–748, doi: 10.1109/ICIP.2016.7532456. [26] M. Gerke and P. Seibold, “Visual inspection of power lines by U.A.S.,” in 2014 International Conference and Exposition on Electrical and Power Engineering (EPE), Iasi, Romania, 2014, pp. 1077–1082, doi: 10.1109/ICEPE.2014.6970074. [27] O. Araar and N. Aouf, “Visual servoing of a Quadrotor UAV for autonomous power lines inspection,” in 22nd Mediterranean Conference on Control and Automation, Palermo, Italy, 2014, pp. 1418–1424, doi: 10.1109/MED.2014.6961575. [28] J. Cui, Y. Zhang, S. Ma, Y. Yi, J. Xin, and D. Liu, “Path planning algorithms for power transmission line inspection using unmanned aerial vehicles,” in 2017 29th Chinese Control And Decision Conference (CCDC), Chongqing, China, 2017, pp. 2304–2309, doi: 10.1109/CCDC.2017.7978899. [29] X. Qin, G. Wu, J. Lei, F. Fan, and X. Ye, “Detecting Inspection Objects of Power Line from Cable Inspection Robot LiDAR Data,” Sensors, vol. 18, no. 4, p. 1284, Apr. 2018, doi: 10.3390/s18041284. [30] L. Matikainen et al., “Remote sensing methods for power line corridor surveys,” ISPRS J. Photogramm. Remote Sens., vol. 119, pp. 10–31, Sep. 2016, doi: 10.1016/j.isprsjprs.2016.04.011. [31] A. Bachrach et al., “Estimation, planning, and mapping for autonomous flight using an RGB-D camera in GPS-denied environments,” Int. J. Robot. Res., vol. 31, no. 11, pp. 1320–1343, Sep. 2012, doi: 10.1177/0278364912455256. [32] A. Bry, A. Bachrach, and N. Roy, “State estimation for aggressive flight in GPS-denied environments using onboard sensing,” in 2012 IEEE International Conference on Robotics and Automation, 2012, pp. 1–8, doi: 10.1109/ICRA.2012.6225295. [33] F. Fraundorfer et al., “Vision-based autonomous mapping and exploration using a quadrotor MAV,” in 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2012, pp. 4557–4564, doi: 10.1109/IROS.2012.6385934. [34] D. Scaramuzza et al., “Vision-Controlled Micro Flying Robots: From System Design to Autonomous Navigation and Mapping in GPS-Denied Environments,” IEEE Robot. Autom. Mag., vol. 21, no. 3, pp. 26–40, Sep. 2014, doi: 10.1109/MRA.2014.2322295. [35] N. Smolyanskiy, A. Kamenev, J. Smith, and S. Birchfield, “Toward LowFlying Autonomous MAV Trail Navigation using Deep Neural Networks for Environmental Awareness,” ArXiv170502550 Cs, May 2017. [36] M. A. Zaidi and R. Colomo-Palacios, “Code Smells Enabled by Artificial Intelligence: A Systematic Mapping,” in Computational Science and Its Applications – ICCSA 2019, vol. 11622, S. Misra, O. Gervasi, B. Murgante, E. Stankova, V. Korkhov, C. Torre, A. M. A. C. Rocha, D. Taniar, B. O. Apduhan, and E. Tarantino, Eds. Cham: Springer International Publishing, 2019, pp. 418–427. doi: 10.1007/978-3-03024305-031: