Increasing number of vehicles and the corresponding increase in traffic on roads has increased the demand of monitoring and controlling of traffic to reduce the number of fatalities. Intelligent transportation systems (ITS) have become an important area of research in the last decade. To introduce such system which can track all the suspicious conditions on the roads and can report the same to reduce the number of accidents and miss-happenings on the roads (Xiao et al., 2020). The most important and the key component of designing an ITS is vehicle detection. Once the vehicle is detected, the information can be precisely used to classify them, analyse the congestion, tracking the vehicles, removing the occlusions, foreign object detection, and detection of suspicious activities and so on (Xiao et al., 2021;Xu et al., 2022).

The study of how to automatically detect and categorise vehicles is a hot topic in the fields of computer vision and machine learning. There are several things -including other cars, buildings, trees, and more -that can obscure a driver's view of their vehicle on the road. Researching how to create algorithms that can accurately recognise and classify partially visible or obstructed cars is difficult. Detecting automobiles in real time is crucial for several uses, including autonomous driving, traffic monitoring, and surveillance. A difficult area of study is the creation of real-time vehicle detection and classification systems. Vehicle identification and categorization algorithms can be hampered by inclement weather. Researching how to make algorithms that can withstand bad weather is difficult. Classifying vehicles is not a simple yes/no question, but rather a multiclass problem. Creating algorithms that can correctly categorise vehicles including cars, trucks, buses, and motorcycles is a difficult area of study. An unbalanced dataset can lower the quality of results obtained by vehicle identification and classification algorithms. It is a difficult research topic to design algorithms that can correct for bias in datasets. The difficulties listed above are only a small sample of the many that have been studied in the field of autonomous vehicle recognition and categorization. By resolving these issues, the performance of these algorithms can be enhanced, and they can be more widely used in practical settings.

The literature review reveals that vehicle detection in traffic videos is difficult because of the dynamic nature of the situations and the large number of possible vehicle types. It can be difficult for vehicle detection algorithms relying on Haar features or HOG descriptors to function in certain settings.

Thus, in this paper YORES an ensemble YOLO Resnet model is proposed. Recent years have seen significant progress in this area thanks to deep learning-based object detection systems like YOLO and Resnet.

The YOLO algorithm is a well-known example of a single-neural-network object detection system. YOLO can detect objects of varying sizes and aspect ratios quickly and precisely. While Resnet is commonly used for image classification and object detection, it is a deep convolutional neural network. Resnet is well-known for its versatility and adaptability, as it can easily handle complicated visual elements and learn from new da-ta.

By combining YOLO and Resnet, we may overcome the shortcomings of both algorithms and improve our ability to detect vehicles. In this research, we propose an ensemble method for vehicle detection in traffic videos by combining the advantages of YOLO and Resnet. The detection of small items or objects with low contrast against their back-ground may be difficult for YOLO. YOLO predicts the bounding box and class of each object using a single grid cell, which may not be precise enough for localizing small objects. Combining YOLO with other object detection model, ResNet, can help alleviate this shortcoming by leveraging the advantages of each.

A further shortcoming of independent object identification models is their potential inability to distinguish between background clutter and actual objects. By combining the benefits of YOLO and REsNet with alternate architectural designs, training data, or input representations, model ensembles can help overcome this restriction. The resulting object detection system may be better equipped to deal with the wide variety of conditions found in the real world.

There are two phases to our ensemble method. In the first phase, coarse object detection is carried out with the help of YOLO. YOLO can swiftly detect the presence of auto-mobiles in an image or video because it has been trained on a vast collection of traffic footage. When cars are identified, YOLO generates bounding boxes that contain those places.

Resnet is utilized for fine-grained detection in the second step. Resnet, which was trained on a more limited set of traffic recordings than YOLO, is able to improve upon the latter's detections by pinpointing the exact position and orientation of the vehicles. Resnet's output is also a set of bounding boxes that are associated with the observed vehicle locations.

In order to arrive at a conclusive detection result, the results from both algorithms are integrated via weighted average. Each algorithm's performance on a validation dataset is used to determine the weights, which can be tweaked to give more weight to speed or accuracy.

The videos are converted to frames for further processing and identification of vehicles. Noise may be present in the frames due to different illumination, weather, and camera calibrations. Filtering techniques are applied during the pre-processing stage and all the frames are converted into a normalized size of 224 × 224 × 3. The details of the complete method are described in the sections below (figure 1).

The traffic scenes to be processed are generally captured by the CCTV cameras installed on roads. These cameras capture the vehicles as a video. The processing of these videos cannot be done directly. Thus, the conversion of the videos to image frames captured at different time frames is required. A video taken over a time interval T may be represented as shown in equation (1).

where 𝑣 𝑇 = Traffic Video recorded at time interval 𝑇 . 𝑓 𝑛 = image frame. 𝑛 = number of frames per second.

The pre-processing of the retrieved video frames is important. As these frames suffer from poor quality due to different capturing conditions. They may also have noise due to the problems in the image sensors. All these will lead to poor results and therefore some pre-processing is required. After pre-processing the data will be ready for input to the model. The main issue is presence of noise. Thus, the input frames are filtered using Butterworth low pass noise removal filter (Basu, 2002) for removing the noise and smoothening the images. The mathematical equation for the same is given in eq. (2).

where 𝐷 0 is the cut − off frequency and 𝐷(𝑥, 𝑦) = √𝑥 2 + 𝑦 2 where 𝑥 𝑎𝑛𝑑 𝑦 are individual pixels of HSI layers obtained in previous step.

Train Test Split

In recent years deep learning has shown very good results for object detection and classifications in image/videos. In this paper, we have used a Resnet-50 network for detecting various vehicles on the road. The network comprises of the convolution layer network which extracts various important features from the image applying convolutions. The second part of the network is feature localization network which comprises of region proposal networks and pooling combined with non-maximum suppressions to detect the bounding boxes around the vehicles. The backbone network used in the proposed work for initial feature extractions is ZF network (Zeiler & Fergus, 2014). The network has very fast training and testing speed and is very useful in designing real time object detections. The network uses small size kernels which maintain even lower-level details in the frames with max pooling. This reduces the time and complexity in network processing.

The second network used is YOLO which is efficient and fast object detection network (Diwan et al., 2023). The network architecture for YOLO is shown below:

1. Input Layer: This layer is responsible for receiving the input video frames (RGB) from the traffic videos.

2. Backbone Network: The EfficientNet design serves as the foundation for the backbone network, which is made up of numerous convolutional layers and includes the following: a. Convolutional layers: The backbone network contains a total of 9 convolutional layers, each with a different number of filters and kernel sizes. b. Bottleneck layers: The backbone network is comprised of 2 bottleneck layers, each of which utilizes a combination of 1x1 and 3x3 convolutional layers to minimize the total number of input channels. c. Depthwise separable convolutions: The backbone network also includes two depthwise separable convolutional layers. These layers make use of a combination of depthwise and pointwise convolutions in order to reduce the number of computations that are necessary for feature extraction. 3. The Neck Network: The neck network is what connects the head network to the backbone network. It is made up of a few convolutional layers and includes the following components:

a. SPP layer: The neck network incorporates a spatial pyramid pooling (SPP) layer, which implements max pooling at many scales to capture features at various granularities of detail. b. Convolutional layers: The neck network also incorporates a number of convolutional layers, which further refine the features that were extracted by the backbone network. 4. Head Network: The head network is the part of the system that is in charge of producing bounding boxes and detecting objects. The head network is made up of a number of convolutional layers, including the following: a. Levels of prediction that are based on anchors: The head network has three levels of prediction that are based on anchors. Each of these layers predicts the class and location of objects by making use of anchor boxes that range in scale. b. Convolutional layers: The head network also includes a number of convolutional layers, which further refine the predictions that were provided by the anchor-based prediction layers. 5. Output Layer: The output layer is responsible for generating the final detection results, which include the category and position of each object that was found.

Let 𝐼 be an input video frame and let 𝑌𝑂𝐿𝑂(𝐼) be the output of 𝑌𝑂𝐿𝑂 on 𝐼, which consists of a set of bounding boxes 𝐵 = {𝑏 1 , 𝑏 2 , 𝑏 3 , … 𝑏 𝑛 }, where each 𝑏 𝑖 = (𝑥 𝑖 , 𝑦 𝑖 , 𝑤 𝑖 , ℎ 𝑖 ) represents the location and size of a detected vehicle.

Let 𝑅𝑒𝑠𝑁𝑒𝑡(𝐼) be the output of 𝑅𝑒𝑠𝑁𝑒𝑡 on 𝐼, which also consists of a set of bounding boxes 𝐵 ′ = {𝑏 1 ′ , 𝑏 2 ′ , 𝑏 3 ′ , … 𝑏 𝑛 ′ }, where each 𝑏 𝑖 ′ = (𝑥 𝑖 ′ , 𝑦 𝑖 ′ , 𝑤 𝑖 ′ , ℎ 𝑖 ′ ) represents the location and size of a detected vehicle.

We can combine the outputs of YOLO and Resnet using a weighted average:

where 𝐵 𝑓𝑖𝑛𝑎𝑙 = {𝑏 1 𝑓𝑖𝑛𝑎𝑙 , 𝑏 2 𝑓𝑖𝑛𝑎𝑙 , 𝑏 3 𝑓𝑖𝑛𝑎𝑙 , … 𝑏 𝑛 𝑓𝑖𝑛𝑎𝑙 } is the final set of bounding boxes, and 𝑤 1 and 𝑤 2 are the weights assigned to YOLO and Resnet, respectively. We can choose these weights based on the performance of each algorithm on a validation dataset, and we can adjust them to prioritize speed or accuracy depending on the application. Both the YOLO and ResNet models produce a significant number of ideas for each vehicle. The abundance of proposals poses a challenge in the process of filtering and identifying a singular bounding box for each vehicle. Therefore, the application of the non-maximum suppression technique is employed to filter the bounding boxes and reduce them to a single box per vehicle. The NMS algorithm takes as input a set of proposal boxes B, their corresponding confidence scores (S), and a user-selected threshold value (N). The filtered proposals (D) are acquired as the resulting output of the method.

The employed loss function in this study is a multifaceted loss function. The losses employed in this context are mean squared error losses, which incorporate the IoU score to quantify the discrepancy between expected and actual values. The loss function comprises three components, namely coordinate loss, confidence loss, and classification loss.

where

(5)

The cross-entropy loss function is defined as the difference between the true probability distribution y and the anticipated probability distribution.

Where 𝒚 𝒊 is the ith element of the true probability distribution y and 𝒚 𝒊 ̂ is the corresponding element of the predicted probability distribution ŷ. The summation is taken over all elements i of the distributions.

When combining YOLO with ResNet, we can use a loss function that is a weighted sum of the losses from both models. For instance, we can compute the loss as a weighted sum of the YOLO and ResNet loss functions by giving each loss term in the YOLO loss function a certain value. Training model parameters can also be updated using a weighted combination of individual model's optimization techniques. The relative success of each model on the validation data will inform the decision of how much weight to give each feature.

YORES ensembles YOLO and ResNet -and that their loss functions, L(YOLO) and L(ResNet), have been assigned weights of alpha and beta, respectively. The ensemble loss function is then calculated as:

Here, alpha and beta are scalar weights that specify how much emphasis should be placed on either of the two loss functions. These weights can be determined by looking at how well each model does on validation data and giving more weight to the model that does better.

Minimizing the global loss function 𝐿 𝑒𝑛𝑠𝑒𝑚𝑏𝑙𝑒 as a function of the model parameters is the target of the optimization. Backpropagation is used to compute the gradients of the global loss function with respect to the model parameters during training, and an optimization technique like stochastic gradient descent (SGD), Adam, or RMSProp is used to update the model parameters.

The loss functions of YOLO and ResNet are combined in the ensembled model by weighting the individual loss functions and then summing them to get the final loss function.

In this section the experiments are discussed. The proposed model is implemented using Python programming language. The Python modules used include keras and tensorflow. Other supporting modules are also used. The model training is done using GPU support as the dataset is very large and training will ot be possible with simple CPU. The dataset contains two subsets localization and classification. Using these datasets an annotated csv file is created. The csv file comprises the bounding box position of each object. This is used as the ground truth for training. The model is trained with 10000 iterations on the selected dataset. After the trained network is fully trained it can identify the vehicles. The non max suppression threshold value is selected as 0.45. The model is then applied to test videos and images. Each detected object shows vehicle bounded by a box. The name of the vehicle also appears on the box. The results of various steps of the proposed method are shown below. Classification results for all categories of vehicles are shown in figure 3.

The experiments are conducted using the publicly available datasets. Numerous publicly available datasets are available for vehicle classes. But in this work one of largest vehicle dataset MIO-TCD (Luo et al., 2018) is used. This dataset is divided into two parts the classification and localization dataset. The localization is used for the object position and classification for vehicle class. The distribution of the MIO-TCD dataset is shown in table 1. The dataset contains vehicles from different field of views, illumination condition and weather. Some of the sample images from the dataset are shown in figure 2.

To quantitatively analyse the performance of the classification and detection method following metrics are being used.

Total Accuracy: The total accuracy demonstrates the percentage of the total number of vehicles currently identified as vehicles.

𝑇𝐶𝐼 𝑡𝑜𝑡𝑎 𝑙𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑣𝑒ℎ𝑖𝑐𝑙𝑒𝑠 (10) where 𝑇𝐶𝐼 = total number of correctly identified vehicles

The dataset which we are using is having different number of images or frames for different datasets we have used another two metrics for rectifying this imbalance namely mean recall and mean precision. These are obtained by taking the average of precision and recall overall category of the vehicles. The method was also compared with other state of the art classification methods. All these methods have used feature extraction followed by a classifier network. The comparative results for all the methods have been shown in Table 3. The graphical representations of the same are shown in figure 4. The average accuracy comparison analysis with different methods is shown in figure 9.

In this research, we offer an ensemble method to the problem of vehicle detection in traffic videos by combining the advantages of the YOLO and Resnet algorithms. YOLO is used for coarse object detection, and Resnet is used for fine-grained detection in our approach. YOLO is used to identify large groups of objects. A weighted average assembly is used to aggregate the results of both of these processes. The YOLO and ResNet loss functions are combined in the ensembled model by first assigning weights to each loss function and then computing the weighted sum of these individual loss functions as the overall loss function. We can effectively combine the strengths of YOLO and ResNet and increase the performance of car detection from traffic videos by improving the ensembled model using the overall loss function. This allows us to effectively mix YOLO and ResNet. Some of the limitations of standalone object detection models can be circumvented with the help of ensembling and YOLO. The proposed approach accurately identifies eleven classes of vehicles, achieving state-of-the-art results. The comparison of the proposed approach with six other methods demonstrates its superiority in terms of accuracy, speed, and robustness. Therefore, the proposed approach has significant potential for practical applications in traffic surveillance and management, such as traffic flow optimization and accident detection. Further studies can investigate the scalability and generalizability of the proposed approach to various traffic scenarios and different environments. Overall, this research contributes to the development of intelligent transportation systems and paves the way for future research in this field.