This paper aims to develop a method that can accurately count vehicles from images of parking areas captured by smart cameras. To this end, we have proposed a deep learning-based approach for car detection that permits the input images to be of arbitrary perspectives, illumination, and occlusions. No other information about the scenes is needed, such as the position of the parking lots or the perspective maps. This solution is tested using Counting CNRPark-EXT, a new dataset created for this speci c task and that is another contribution to our research. Our experiments show that our solution outperforms the stateof-the-art approaches.
This paper is motivated by the need to address the challenging real-world counting problem related to the estimation of the number of vehicles present in a park area, using images captured by smart cameras. The visual understanding of the collected images must face many challenges, perhaps common to all the counting tasks, like variations in scales and perspectives, the inter-object occlusions, the non-uniform illumination of the scene, and many others.
In order to address these challenges, we propose a deep learning-based approach that is able to accurately count cars in images without any extra information of the scenes, like the position of the parking lots or the perspective map. The latter aspect is a key feature since in this way our solution is directly applicable in unconstrained contexts.
To validate our approach, we also built a dataset, called Counting
CNRParkEXT dataset, collecting images from the parking lots in the campus of the
National Research Council (CNR) in Pisa. The images are taken by nine di erent
cameras in challenging conditions since they are captured from di erent
perspectives, they present di erent illuminations and many occlusions. The result
of the proposed methodology signi cantly outperforms the ones obtained using
the state-of-the-art baseline methods.
Objects counting has been tackled in computer vision by various techniques,
especially for the estimation of the number of people in crowded scenes. Following
the taxonomies adopted in [
Counting by detection is a supervised approach where a sliding window
detector (i.e. a classi er that is slid over the entire image) previously trained is
used to detect objects in the scene. This information is then used to count the
number of objects. In the monolithic detection the classi er is trained in order
to recognize the whole object we want to detect [
Counting by clustering tackles the counting problem in an unsupervised way.
A clear advantage is that such an approach does not need to be trained and it
is out of the box. However, the counting accuracy of such fully unsupervised
methods is in general limited. The clustering by self-similarities technique relies
on tracking simple image features and probabilistically group them into
clusters, like in [
Counting by regression is a supervised method that tries to establish a direct mapping (linear or not) from the image features to the number of objects present in an image without explicit object detection or tracking. Since it does not rely on a speci c classi er or model previously trained, it is more robust to occlusions and perspective distortions.
Finally, counting by density estimation is a supervised technique that
extends in some way counting by regression approach, introduced in [
The extraction of suitable features is a crucial operation for all the four
approaches that have been described. Since, in general, handcrafted features
su er a drop in accuracy when subjected to challenging situations (variances
in illumination, perspective distortion, severe occlusion, etc.), most
state-of-theart counting methods employ deep-learning approaches, in particular exploiting
Convolutional Neural Networks (CNNs), in order to extract suitable task-speci c
features automatically. Some works that use CNNs are [
A contribution of this paper is the creation of "Counting CNRPark-EXT", a
dataset of roughly 4,000 images containing more than 79,000 labeled cars. This
dataset is based on the CNRPark-EXT dataset, presented in [
The CNRPark-EXT dataset is speci cally designed for parking lot occupancy detection and it is not directly usable for the counting task, since each image, called patch, contains one single park space labeled according to the occupancy status of it, 0 for vacant and 1 for occupied. Since the purpose of this work is to count cars present in an image, the information needed to build the ground truth are full images (and not patches), the total number of vehicles considered in an entire scene, and, at least for some techniques, the locations of these vehicles. Therefore, starting from the CNRPark-EXT dataset, we summed up the occupancy status of the patches belonging to a same full image (note that a car space is occupied if and only if a car is present in the car space), obtaining the total number of cars considered in the whole scenes (i.e. obtaining a perimage ground truth). Only bounding boxes corresponding to occupied spaces are considered, identifying cars to be counted (i.e. obtaining a per-object ground truth). Table 1 reports the composition of Counting CNRPark-EXT dataset.
A new vehicles counting solution using Mask R-CNN
Our solution is based on Mask R-CNN [
As a starting point, we considered a model of Mask R-CNN pre-trained on
the COCO dataset [
The rst step has been the creation of a suitable labeled training set. In the case of Mask R-CNN, these labels correspond to masks and bounding boxes. As mentioned before, since the labels of our dataset are bounding boxes (perhaps not very accurate, since they localize parking lots and not the cars), we needed to add the masks on the vehicles to be detected in order to make Counting CNRPark-EXT dataset useful for our purposes. Mask creation is a very timeexpensive operation - for each car in each image, we should associate the pixels that model the vehicles with a label. Since in the dataset we have hundreds of cars, this problem has been solved by taking advantage of the output of the pretrained model of Mask R-CNN. Since this network produces in output accurate masks localizing objects, the idea is to save these masks automatically generated, parse them, and then manually add some of the remaining masks for the cars that were not detected.
First, we randomly selected a subset of our training set (about the 10%), fed the previous pre-trained Mask R-CNN model with these images, and saved the vertex coordinates of the polygons surrounding the masks localizing objects that the network produces in output along with the category associated with them. Next, a further analysis of these saved masks has been performed, changing the wrong associations between some objects labeled with categories that are not possible in our context (like airplane and boat ), and that were instead vehicles we want to detect. In this way, we forced the network to learn to recognize these objects as cars and not as other wrong objects. Finally, we manually add some of the remaining masks localizing vehicles that were not automatically detected. At this point, we retrained the network using this new mask-labeled and taskspeci c dataset, frozen the weights of the backbone, and saving the new weights of the head after a few epochs. Figure 1 reports the pipeline of the described approach. In this section, we present the methodology and the results of the experimental evaluation. For testing purposes, we have used the test subset of the Counting CNRPark-EXT dataset.
Evaluation Metrics Following other counting benchmarks, we use Mean Absolute Error (MAE ) and Root Mean Square Error (RMSE ) as the metrics for comparing the performance of our solution against other counting approaches present in literature. MAE is de ned as follows: while RMSE is de ned as:
M AE = 1 XN cgt
j n
N n=1 v RM SE = tuu N1 n=1
N X(cgnt cpred n
j cpnred)2 (1) (2) where N is the total number of test images, cgt is the actual count, and cpred is the predicted count of the n-th image. Note that as a result of the squaring of each di erence, RMSE e ectively penalizes large errors more heavily than small ones. Then RMSE should be more useful when large errors are particularly undesirable.
The above two metrics are indicative of quantifying the error of estimation
of the objects count. However, as pointed out by [
M OE = 1 XN jcgnt cpnredj N n=1 num slotsn (3) where num slotsn is the total number of parking lots in the current scene. In the next, this evaluation metric is expressed as a percentage.
Comparisons with the state of the art We have compared our vehicles
counting solution against the method proposed in [
The main drawback of such an approach is that the locations of the monitored objects of a scene must be known in advance. This technique fails if it is applied on a new camera added in the car park, just because it does not have the knowledge about where car slots are located: a preliminary annotation of the new camera and a new train of the network are then mandatory operations. So, this issue makes this car counting method not directly applicable in unconstrained contexts.
On the other hand, as already mentioned before, our counting approach is able to count vehicles without any extra information about the parking lots locations. Nevertheless, results using this classi cation approach applied to the Counting CNRPark-EXT dataset can be used as a baseline and as a basis of comparison.
Results The results of the experimental evaluation are reported in Table 2. Note that our solution not only outperforms the original pre-trained Mask-RCNN, but also it works better than the state-of-the-art approach using mAlexNet, considering all the three performance metrics.
Since counting errors could be due to di erent weather conditions (since di erent weather conditions might produce signi cant di erent illuminations of the scenes) and/or to di erent camera views (di erent camera views correspond to di erent perspectives), we perform a further analysis of the results by dividing them according to the number of cameras and to the weather condition they belong to. Figure 2 reports the errors scatter-plots of the errors. Note that, in general, our solution tends to underestimate the number of vehicles present in the scene, but it responds well to perspective and illumination changes, having only a small performance decrease considering frames belonging to camera nine. In this paper, we presented an e cient solution for counting vehicles in parking areas that exploit deep Convolutional Neural Networks (CNNs) to detect vehicles present in challenging scenes. Our proposed methodology does not require any manually entered information about the parking lots locations, allowing a simple 'plug-and-play' installation. The results outperform the ones obtained using the state-of-the-art baseline methods.
As a further contribution, we collected Counting CNRPark-EXT, a dataset containing images of real parking areas captured by nine cameras, with di erent weather conditions and perspective views.