Considering the cost deployment problem of trafic recognition algorithms, this paper considers YOLOv4 as the base architecture. The lightweight DenseNet is used as the backbone feature extraction network, and efective channel attention (ECA) and Adaptive Spatial Feature Fusion (ASFF) are used to enhance the PANet structure with attention-guided fusion. The weight ratio of the loss function is optimized and the mosaic method is used for training enhancement. The results show that the proposed algorithm improves both the detection accuracy and detection speed as well as reduces the number of parameters by 64. The research results provide some reference value for the trafic construction of smart cities.
As the pioneer of one-stage target detection, the YOLO series algorithm innovates on the detection
principle of the Faster R-CNN series by abandoning the RPN approach and using regression to obtain
the coordinate information of the bbox. YOLOv1, an algorithm that uses an end-to-end identification
approach, is known as the one-stage target detection algorithm. This algorithm was quickly deployed in
many real-world projects due to the dramatic increase in detection speed, and was even used in military
devices. A large number of one-stage target detection algorithms have also emerged since then, and
these algorithms have evolved through iterations in pursuit of faster and more accurate recognition
results [
Bochkovskiy et al. [
The YOLOv4 algorithm consists of the CSPDarknet53 backbone network, SPPNet, PANet feature fusion network and the YOLO-Head detection head module that is used in YOLOv3. Its network structure is shown in figure 1.
In this paper, we further improve the training and inference speed of the one-stage detection algorithm by modifying the backbone network of the model, based on the YOLOv4 algorithm, and improve the model structure using the attention mechanism and feature fusion module to enhance the detection performance of the algorithm.
Feature Pyramid Representation (FPN) is a common approach to address the challenge of scale variation in target detection. Its structural layer design allows the model to more fully utilize the feature information extracted from the backbone network.
Various FPNs are designed to maximize the utilization of the multi-scale feature maps from backbone,
and its optimization leads to significant performance improvement of object detection. Therefore, the
algorithms in this paper work in concert with the fusion of PANet and ASFF to enhance the reuse and
extraction of feature maps and avoid the loss of efective information [
The attention mechanism focuses on local information while suppressing distracting information. Attention mechanisms have made important breakthroughs in recent years in areas such as image and natural language processing, and have widely demonstrated their efectiveness in improving model performance.
From a mathematical point of view, the attention mechanisms provides a weight-based model to perform operations. The attention mechanism uses the network layer to calculate the weight values corresponding to the relevant feature maps, and then applies these weights to the feature maps, so that the feature maps with a large role in extracting information become somewhat more influential on the overall. With respect to the content of interest, attention mechanisms can be split into three types: channel attention mechanism, spatial attention mechanism, and mixed spatial and channel attention mechanism.
The Spatial Transformer Network (STN) [
Sequeeze and Excitation Net (SENet) [
Convolutional Block Attention Module (CBAM) [
The backbone network is replaced with the lightweight network DenseNet-121, and the rest of the architecture is optimized on the basis of YOLOv4.
The main components of the DenseNet network are both the dense blocks and transition layers.
The dense block is composed of several bolt necks. Each block uses the same number of output channels, and then uses a loop to connect the input and output of each block in the channel dimension. The structure of bolt neck is shown in the upper part of figure 3. Each bottle neck contains two convolutions, the first one is a 1*1 convolution, which has 4 output channels. Here, is a feature map growth factor, which is the number of feature maps contributed by each bottle neck. The second 3*3 convolution has output channels. Finally, the input of the module and the output of the 3*3 convolution are concat stacked to obtain the overall number of output channels of the module as ′ + .
The dense block structure is shown in the middle part of figure 3, which consists of several bottle neck. If the number of input channels of the whole dense block is 0. Since the output of Bottle Neck stacks the output and input of the final convolutional structure in its interior, the number of feature channels will be increased by for each bottle neck that passes through it. Therefore, the number of ifnal output feature maps of a dense block composed of bottle neck is 0 + .
By looking at the dense block structure, it can be seen that the input of each bottle neck is a stack of all the outputs of its preceding layers. This essentially densely connected network structure is the reason why DenseNet can achieve good results.
The transition layer is used to control the model complexity, and its structure is shown in the lower part of figure 4. Since the number of channels increases with each dense block connection, using too many would result in an overly complex model. Therefore, the transition layer first reduces the number of channels by a 1*1 convolution layer, and then to compress the height and width of the feature map, an average pooling layer with stride=2 is used for downsampling, which further reduces the model complexity.
To ensure the detection accuracy of the model while performing lightweight optimization of the model, this paper will intersperse the attention mechanism module in the network structure.
To keep the balance between model complexity and performance, this paper refers to an efective channel attention module (ECA) that contains only a small number of parameters while delivering significant performance gains.
SE-Net is the basis of ECA-Net optimization and its structure is shown in figure 4(a). Global average pooling is first performed separately for each input channel, followed by two fully connected layers using diferent activation functions. This computational process causes the channel features to be mapped from high to low and then to high dimensions. This dimensionality reduction operation reduces the complexity of the model, but it also hinders the correspondence generated between weights and weights, which may result in the loss of critical information.
Empirical data show ECA-Net, by observing SE-Net and improving it, that it is important to avoid dimensionality reduction when learning channel attention and a proper cross-channel interaction can increase the complexity of the model only slightly while maintaining performance. Its structural design is given in figure 4(b).
On the left is the feature of the original input image, which is first subjected to global average pooling
(GAP) [
In order to avoid the consumption of large arithmetic resources due to manual adjustment, the size of the parameter can be generated adaptively by a function with the convolution kernel : where || denotes the odd number of -nearest neighbors, is set to 2, and is 1. From the equation, it is = () = ︂⌊ log2() + ⌋︂
odd clear that the communication range of the high-dimensional channel is longer, while the communication range of the low-dimensional channel is relatively contracted.
In this paper, three ECA layers are inserted at the connection between backbone and neck of the model to avoid dimensionality reduction while better bridging the two components, making the feature transfer of the model more eficient and preventing the disappearance of feature information. At the same time, the ECA layer allows the model to focus on more critical features and suppress unnecessary features, thus ignoring the interference brought by the image background, which enhances the accuracy of detection for the model even further.
ASFF can further enhance the extraction capability of PANet and can fuse the information of multiple feature layers simultaneously. Its core idea is to adaptively adjust the spatial weights of each scale features in fusion by learning. Its underlying structure is shown in figure 5.
The ASFF-3 is an example of a convolution with a convolution kernel of 3*3, a step size of 2, and a padding of 1. The 2 is scaled down to the same value as 3 with equal number of channels, and is denoted as level_1_resized. The number of channels and dimensionality of level_1_resized, level_2_resized, and 3 are the same. Finally, level_1_resized, level_2_resized, and 3 are multiplied by , and , respectively, and the values are summed, and the number of channels is adjusted by a ifnal convolutional layer to obtain a new feature layer with multi-layer perceptual field fusion. The expression is as follows.
= · 1→ + · 2→ + · 3→ (2) where represents the new feature map of a layer obtained by ASFF, , , and represent the weight parameters learned through the three feature layers, and + + = 1 is guaranteed by the Softmax function.
The loss function based on the YOLOv4 algorithm contains three components: confidence error conf,
classification error cls, and regression frame prediction error loc. Among them, the confidence error
and classification error continue the design idea of YOLOv3 [
− )︁ log(1 − )]︁ − )︁ log(1 − )]︁ (3) (4) (5) (6) cls = − ∑︁ obj ∑︁ {︁ () log [︁ ()]︁ + [︁1 − ()]︁ log [︁1 − () ]︁}︁ ∈classes CIoU(, ) = IoU(, ) − 2(ctr, ctr) 2 − where: 2 is the number of grids, is the number of prediction frames in each grid, obj and noobj are the indicator values of the prediction frames containing and not containing the target, is the confidence true value, is the prediction confidence, noobj is the penalty weight factor, () is the actual probability that the target in the cell belongs to category , () is the probability that the prediction is of category , IoU(, ) is the intersection ratio of the predicted frame to the real frame , 2(ctr, ctr) is the Euclidean distance between the center point of the predicted frame and the real frame, is the diagonal distance of the minimum closed region containing both the predicted and real frames, is the balance adjustment parameter, is the parameter measuring the consistency of the aspect ratio.
The regression frame prediction error loc is typically defined using the CIoU loss: 2 =0 =0 loc = ∑︁ ∑︁ obj(1 − CIoU(
, ^ )) where represents the predicted bounding box and ^ represents the ground truth bounding box.
In order to balance the loss sensitivity of diferent detection scales, in this paper, the three prediction
heads in the network structure are multiplied with diferent weights when calculating the total loss.
The weights assigned to Yolo Head1, Yolo Head2 and Yolo Head3 are 0.4, 1.0 and 4.0, respectively [
6.1.1. Dataset In this paper, a variety of datasets will be used for performance evaluation.
1. The RSOV dataset is published by Brno University of Technology and consists of three subdatasets with diferent viewpoints, namely the rear view shot dataset, the eye level view shot dataset, and the unconstrained shot dataset, each of which contains 5000 images of vehicles with annotations. Thus, the dataset has a total of 15,000 images containing information of 41 diferent brands and categories of vehicles. In this paper, we divide the dataset according to the ratio of 8:1:1, and finally get 12,000 training sets, 1,500 validation sets and 1,500 test sets. 2. BIT-Vehicle dataset is captured by two cameras at diferent times and locations, and these images vary in terms of lighting conditions, vehicle color, and camera viewpoint. All vehicles in the dataset are classified into six categories: Bus, Microbus, Minivan, Sedan, SUV, and Truck. The dataset has a total of 9850 images, and the dataset is divided according to the ratio of 8:1:1, resulting in 7880 training sets, 985 validation sets and 985 test sets. 3. To verify the detection generality of the proposed algorithm, the classical multi-category dataset PASCAL VOC is used. The PASCAL VOC Challenge is a world-class competition in computer vision covering several sub-tasks such as classification of images, detection and segmentation of targets, etc. VOC2007 and VOC2012 are two classical benchmark datasets publicly provided by the competition, including a total of 20 categories including people, airplanes, and cars, and each version of the dataset is produced in a uniform manner. In the paper, we use in total 16,551 images of trainval data from VOC2007 and VOC2012 as the overall dataset, which is randomly partitioned according to the ratio of 0.81:0.09:0.1, resulting in 13,405 training sets, 1,490 validation sets and 1,656 test sets.
Since there are large diferences in the number of samples and uneven distribution of some images in some datasets, mosaic data enhancement is performed on the dataset before model training. The operation process is shown in figure 6.
The mosaic data enhancement method first takes out a batch of data to be processed from the dataset, then randomly uses four images to be scaled or shifted in diferent proportions, placed in the direction of the four corners of the rectangle, crops the excess part of images that exceed the specified input size, and finally gets a new image as training data. The mosaic method for data preprocessing not only enhances the diversity of data and enriches the image dataset, but also improves the batchsize in disguise and enhances the eficiency of the model.
1. Evaluation metrics for model inference
For the target detection task, the precision , recall , and mean average precision (mean Average Precision) are commonly used as evaluation metrics for model identification. The calculation of the relevant metrics is given below.
a) Precision and recall .
The precision is the ratio of correct predictions to the total number of predictions. It is one of the simplest metrics. Recall calculates the ratio of the number of predicted positive cases to the total number of positive case labels. They are calculated as follows:
= , (8)
+ where denotes the number of positive samples correctly identified as positive, denotes the number of negative samples incorrectly identified as positive, and denotes the number of positive samples incorrectly identified as negative.
=
+ b) and .
Average Precision ( ) is the area limited by the precision-recall curve. The better a classifier is, the higher the value. It is used to evaluate the detection accuracy of a class, and is calculated as follows:
0
The mean Average Precision ( ), which is the average value of multiple categories
of . The has a size in the interval [
=1
FLOPsconv = ℎout × out × (2 × ℎ × × in − 1) × out where ℎout and out are the height and width of the output feature map. The term (2 × ℎ × × in − 1) accounts for multiplication-addition operations for each output element. For a fully connected layer: where in is the number of input neurons and out is the number of output neurons. The total computational complexity of a neural network model can then be expressed as: fc = (in + 1) × out FLOPsfc = (2 × in − 1) × out
Total FLOPs = ∑︁ FLOPs =1 2. Evaluation Index of Model Parameters
In a deep learning model, the size of the number of parameters determines to some extent the depth of the model, the speed of inference and even the detection accuracy. A large deep learning architecture is often accompanied by high accuracy because it is closer to the neuronal composition of the human brain. However, a large number of parameters also means a sacrifice of inference time and response speed. Therefore, a small number of parameters is important for deploying deep learning models to embedded devices or platforms, and for handling large numbers of concurrent requests.
The metrics of floating point operations (FLOPs) is referred to as the number of float-point operations. It is understood as the number of computations, and the base unit is . It can be used to measure the complexity of an algorithm/model.
Params is referred to as a total number of parameters to be trained, and the base unit is . Params = ∑︁ =1 where is the total number of layers in the model and is the number of parameters in layer . For a typical convolutional layer, the number of parameters can be calculated as: conv = (ℎ × × in + 1) × out where ℎ and are the kernel height and width, in is the number of input channels, out is the number of output channels, and the +1 term accounts for the bias parameter for each output channel.
For in a convolutional layer: where FLOPs is the number of floating point operations in layer . (9) (10) (11) (12) (13) (14) (15) (16)
Model eficiency can be characterized by the ratio:
Eficiency =
Accuracy
FLOPs × Params This metric helps quantify the trade-of between model performance and computational resources required.
In the paper, we use migration learning to speed up the training of models. Migration learning is used
to help train a new model by migrating the weight parameters of an already trained model to a new
one. The basis of this approach is that most of the data or tasks are correlated, and by sharing some of
the parameters of the pre-trained model to the new model to be trained, the training process of the new
model can be significantly accelerated and optimized for the purpose of saving computational resources
[
In deep neural networks, the previous convolutional layers generally learn shallow features with generality, while the later convolutional layers learn more targeted, higher-level abstract features for the current training target. Freezing some of the network layers first can speed up the training and also prevent the weights from being corrupted in the early stage of training. Migration learning can also efectively avoid the problem of poor generalization of the model due to the existence of local minima in the objective function. The migration learning strategy in this paper is as follows. 1. Selecting publicly available DenseNet weight files that have been trained through Imagenet or other large datasets as the parameter source for pre-trained weights for migration learning. 2. Load the weight files into the backbone network of this paper’s model, and then freeze the backbone network without participating in back propagation. The other unfrozen network layers are trained with a certain number of epochs to perform gradient updates. 3. After a certain number of epoch iterations, the frozen layers are unfrozen and all network layers are involved in the backpropagation update to finally obtain the appropriate parameter matrix and bias vector.
The experiments in this paper are conducted under Linux with Intel(R) Xeon(R) CPU E5-2678 v3 @ 2.50GHz processor, 100GB RAM, NVIDIA GTX1070Ti graphics card, and Pytorch 1.8.0 framework for model training and testing. The training parameters were set as shown in table 1.
Since the framework of this paper is inspired by the structure of YOLOv4, this paper mainly uses YOLOv4 as the comparison object to test the improvement results.
1. Training process comparison. (17)
To show the convergence of the model, the YOLOv4 algorithm is compared with the proposed algorithm in terms of the loss values at training time. Since the initial training loss values are large, the curve generation will disturb the overall display, so the values of the first 5 epochs are removed from the loss curve graph. The final loss curves of the two models can be seen in figure 7.
On the figure, we can see that the trend of the loss value tends to be smooth in the last 10 epochs, and there is almost no change in the magnitude, and the final convergence for the proposed algorithm is smaller than that of the original YOLOv4 model under the same loss function calculation for both models.
2. Model complexity comparison
The FLOPs and Params introduced in the previous section are used here to measure the complexity of the models.
According to the diferent model components and usage methods, the experimental groups are divided into four groups for comparison in this paper. The group marked as A1 is the original YOLOv4 algorithm; the group (A2) is based on YOLOv4 with the addition of an ECA layer in addition to replacing the backbone network with DenseNet; while the third group (A3) difers from A2 in that the ECA layer is turned into an ASFF structure; and finally, the fourth group (A4) has the proposed algorithm, with the replacement of the DenseNet backbone network and adding both ECA and ASFF layers. The results of each experimental group are given in table 2.
As we can see, the original YOLOv4 algorithm has the highest complexity in the table, thus it takes longer time in training and inference, and the final generated weight file takes up more space. The comparison between A3 and A4 shows that the ECA attention structure not only works well, but also increases the computational pressure minimally; by comparing A2 and A4, it can be seen that the ASFF structure has a certain fraction on both FLOPs and Params when the computation of the ECA layer is known to be tiny. By comparing A1 and A4, we can see that the computation of the proposed algorithm in this paper is only 51% of that for YOLOv4 and the number of parameters is only 36% of that of the original one, so the proposed algorithm can significantly reduce the usage cost of the model and is beneficial to the deployment of vehicle detection algorithms in practice.
3. Comparison of detection efect
In terms of detection efectiveness, YOLOv4 is used as baseline to compare with the proposed algorithm on a variety of datasets. The results are shown in table 3. As it can be seen from the table, the proposed algorithm has an improvement efect on diferent datasets compared to Baseline. In the tasks and datasets related to vehicle detection, the RSOV dataset improves by 3.53% and the BIT-Vehicle dataset improves by 2.46%. This confirms that the proposed algorithm has good applicability in the vehicle detection task. Meanwhile, the mAP of the PASCAL VOC dataset improves by 2.86%. This confirms that the proposed algorithm has good generalization ability and still performs well.
Algorithm YOLOv4 Our
RSOV
In figure 8, the recognition efects and detection heat maps of the two algorithms under the generic task are shown. The comparison between D1 and D2 shows that both algorithms have excellent performance in detecting people, however, D1 has the situation of missing detection for the obscured transportation, and it can also be seen from the heat map E1 that the YOLOv4 algorithm does not pay much attention to the target location of the transportation in the missing detection area. The recognition ability of D2 has been greatly improved. On E2 we see that the attention of the proposed algorithm to the missed detection region of D1 has been well improved.
To further validate the downstream detection task and the proposed algorithm, the proposed algorithm is compared with other mainstream algorithms in with respect to detection efect, complexity and various other indicators. The experimental conditions and parameter settings are the same as in section 6.1.5 of in this paper, and the data sets are selected as RSOV for vehicle detection tasks and VOC data sets representing generic detection tasks, and the comparison results of various algorithms are illustrated in table 4. As can be seen, the proposed algorithm outperforms other algorithms on a variety of data sets. The YOLOv4-MobilenetV3 network, which combines the MobilenetV3 backbone network with YOLOv4, has the simplest model with a low number of parameters, but its mAP has a significant gap compared with other algorithms, making it dificult to meet the requirements for accurate vehicle recognition in trafic scenarios. EficientDet-d3 and YOLOv4-MobilenetV3 have the same level of counts and excellent detection, but the algorithm takes longer time to train, on average 2-3 times longer than other one-stage algorithms, so it is not suitable for scenarios with strict speed requirements. YOLOv3 and YOLOv4 are both classic algorithms of the YOLO family, and although their detection capabilities perform well, the model parameters of both algorithms are relatively large, which can bring hardware expenses at the practical deployment level for trafic vehicle detection tasks. The more efective retinanet is reduced in the number of parameters compared to the former, but its computational efort is significantly aggravated, and the detection efect is also a certain distance from the proposed algorithm. Therefore, the above comparison results show that the proposed algorithm has a balance of computation, number of parameters and detection efect, not only has excellent detection efect, but also has advantages in training dificulty and model deployment cost compared with other algorithms.
Ablation experiments are mainly used to analyze the degree of influence of diferent components on the whole model. To highlight the efectiveness and synergy of the improvement points of the proposed algorithm, ablation experiments are conducted by using some of the improvement points in this paper. The dataset is selected as RSOV for vehicle detection task and VOC dataset representing generic detection task, where the experimental configuration and parameter settings for the ablation experiments are the same as in section 6.1.5 of this paper.
In this ablation experiment, the component modules are classified as follows: DenseNet backbone network module, ECA attention module, ASFF feature fusion module, and mosaic data enhancement module. The mAP performance of each of their groups is shown in table 5. In table 5, “✓” indicates that the method is used and “–” indicates that the method is not used.
Firstly, we can see there is a considerable improvement in the detection eficiency of the target when all the improved modules are working in concert. From the experimental data of G1 and G2, it can be seen that the detection network using DenseNet lightweight backbone has good feature extraction ability for a variety of datasets based on migration learning. Comparing G4, G5 and G6, it can be seen that the obvious enhancements to mAP are the ASFF module and the mosaic data enhancement method. These two methods enhance the utilization of feature information and the generalization ability of the model, especially the synergistic efect of the two makes the enhancement more obvious. As can be seen in G3, the enhancement efect of the ECA attention module is small in percentage, however, its ability to refine the efective information in the channels allows it to enhance the mAP to a new superior value when used in combination with the ASFF module and the mosaic method. Thus, this ablation experiment is a good demonstration of the efectiveness of each component module of the proposed algorithm.
This paper focuses on the one-stage target detection method which has higher requirements for detection speed and deployment cost. Therefore, the proposed method uses YOLOv4 as the base architecture, and significantly reduces the number of parameters in the model by replacing the DenseNet, which has excellent performance, as the backbone feature extraction network; reconstructs the existing FPN network module, uses the ECA attention structure for the transition and transfer of feature information between backbone and neck, and adds the information cross-fusion function before the final detection layer of the network of the ASFF structure; while optimizing in terms of loss function and image preprocessing. The studies of the eficiency of the proposed method are carried out on RSOV, BITVehicle and VOC datasets.
The training process converges faster with a lower value of the loss function for the proposed method. Comparison of complexity between the proposed method and the basic YOLOv4 shows the almost twofold decrease in complexity and the number of parameters is reduced by 64%. The detection accuracy in various datasets with diferent degrees, for example, the mAP reaches 98.70% on the test set of RSOV dataset. The research results provide some reference value for the trafic construction of smart cities.
For the data augmentation algorithm and trafic information recognition algorithm proposed in this article, the expansion and optimization of the dataset can be considered. By expanding and optimizing the dataset, the accuracy and generalization of trafic information recognition can be improved. For example, images of diferent vehicle types can be added to enrich the dataset of trafic scenes, improve data diversity, and more data augmentation techniques such as scaling and random cropping can be used to increase data volume. We also considered the deployment possibility of models in embedded devices in vehicles, and therefore strictly controlled the complexity of algorithms.
Recently new versions of YOLO appeared. In our future studies, we will focus on implementing the developed algorithm in modern versions of YOLO.
Declaration on Generative AI: The author have not employed any generative AI tools.