-In this paper we propose a novel approach to detecting road lanes from video stream in bad-light road scenarios. The main focus of this article is given to the introduction new image binarization method in non-common color space followed by improved density hierarchical clustering algorithm called HDBSCAN. These techniques allow to detect lane boundaries even in low-light scenarios with robust and parameter-free setup.
Keywords—Lane detection, Computer vision, HDBSCAN, DBSCAN, Image Thresholding
I.
INTRODUCTION
The aim of this paper is to design low-cost in terms of processing time pipeline for robust road marking detection in low-level of light road scenarios.
Self-driving cars are actively introduced into people live s. Their number and complexity of software of on-board computers are increasing [1]. Cars of only one Waymo company managed to drive twenty million miles in self-driving mode 1. Nowadays most of companies are relying on solutions with deep neural networks for perception module of the car [2]. Moreover most of them are using radars and lidars, which allow to perceive the environment even in dim and dark road scenarios unlike to usual camera [3]. Despite the fact that these approaches are dominant in the field, they have major limitations that restrict them from full implementation in the industry and interfere with scalability to more users [4].
There is a need for training deep neural networks on a powerful machine equipped with a lot of video cards for achievement of good precision and recall of an output detections. Furthermore, even after successful training and deploying such big model developers need to install high performance computers on the car because real-time executi on is essential in the case of self-driving car. This solution i s more difficult to scale, not to mention the trend to small-siz e components and reducing their cost. Interest in single-boa rd computers is gradually rising because their best qualities — compactness and price [5].
A lot of disputes in the community of self-driving cars are ongoing right now. Some researchers consider advantages and disadvantages of using cameras or lidars [6]. The experience of 1K. Wiggers, ”Waymos’ autonomous cars have driven 20 million mile s on public roads”, 6 Jan 2020, https://venturebeat.com/2020/01/06/waymosautonomous-cars-have-driven-20-million-miles-on-public -roads/ Tesla company shows, that only-camera solution is possible 2. Main disadvantage of lidar is its price, that is comparable in some cases to the price of the car itself. And most of the solutions require a lot more than one lidar on the car (up to 6 small and big lidars, that are mounted on various sides of the car). Some researches and forecasters insist that the price of lidars will eventually fall down3. However, other ones compare camera with human visual cortex, which can reliably identify and detect distance to various objects4.
And this is our motivation why we focus on developing system, that detects road lane markings with the cameras. We believe that such a solution should meet the following requirements: • • • • • • •
System should be cheap for maintaining and producing for the purpose of high scalability.
System should be robust to rapidly changing light environment.
System should be fast for not powerful computers and shouldnt’ require a lot of computational power.
Image processing in color space CIE L*a*b, that is decreasing light impact on the scene and image.
Using HDBSCAN — more recent method of density clustering instead of DBSCAN.
Combined together, these methods provide a robust pipeline with few parameters that need to be configured. Thus, the time for configuring is reduced, which seems very useful. By the term pipeline here and after we mean a set of data processing elements connected in series, where the output of one element is the input of the next one5. LIDAR:
That?”, description,
The paper is organized as follows. Section 2 is giving brief review of other papers in the field. Section 3 describes proposed approach in details. Section 4 compares proposed approach with other ones. Finally, Section 5 gives the conclusion.
II.
RELATED WORK
In recent years neural networks have become a common tool in image processing tasks. In the field of driving an autonomous car, neural networks are often used to detect road markings, obstacles and road signs. The article [7] describes a method for detecting road marking lines based on neural networks. The authors provide experimental data indicating a high accuracy of detection of road marking lines and a high image processing speed. However, the experiments were conducted on equipment, the cost of which is approximately equal to $2,000, and such a price may not be acceptable. The article [8] presents experimental data for several algorithms for detecting lanes using neural networks. All algorithms presented in this article use either expensive or specialized equipment for image processing. This can lead to a significant increase in cost, or narrow the scope of the possible application of the system.
Materials used for carriageway marking lines often have
bright colours which differ a lot from the rest the roads’
surface. Consequently, some algorithms for road marking line
detection utilize this quality and use prior knowledge of
colours for thresholding [9], [
Some road lane detection methods use density clustering
for finding lane on the road image. For example, authors in
paper [
Aside from this paper, clustering in the lane marking scenario is used in the work [15]. In this paper authors use simple hierarchical clustering and new method of post-processing clustering results. For every calculated cluster they calculate the slope line of it, after that they count the intersections of these slopes with other clusters. The clusters with the most number of intersections are used later. This improvement deals with filtration of clusters that are elongated more on the y axis instead of x axis and utilized the geometrical feature of almost every road marking, which are located mostly one on the top of another on one line. This improvement work well on straight road scenarios, nonetheless it can fail on turns or curb road environment.
III.
In this section we examine our algorithm in details. The full block diagram of the approach can be found on Fig. 2
Result: Clustered image points initialization; while not end of video do
Read current color frame; Convert image into CIE Lab color space; Normalize L-channel using MIN-MAX method; Calculate mean and standard deviation of the L-image; Calculate threshold value; Binarize L-channel; Image < − Select pixels from initial color frames, where thresholded image != 0; Initalize HDBSCAN with selected parametrs; Downscale image; Cluster Labels, Labels Probabilites < − Clusterize image with HDBSCAN algorithm; Discard labels with low probability(< 0.75); Mask labels to image;
Resize image with labels to initial size; end
It is necessary to define two assumptions in order to choose a threshold value of binary mask with road marking from an image.
Assumption 1: Road marking lines takes about 5% of an image.
Assumption 2: Road marking lines brighter than majority of objects on a road image (road surface, roadside, cars, etc.).
Adaptive thresholding
Combining thresholded binary mask and color mask
HDBSCAN hierarchical clustering
Result
The image with road segment is captured by camera in RGB color space. In order to reduce noise on the image we preprocess the image by applying Gaussian blur [16]. In experiments the kernel with size 15 was chosen to filter highfrequency noises on the image.
The next step is converting the image from RGB to desired color space. We propose to use one of the perceptually uniform color space — CIELab [17]. In this color space each pixel is encoded with three values: L, a and b. L describes the brightness of a pixel, in other words, characteristic of luminance. a and b values describe chromatic characteristics, from green to red and from blue to yellow, respectively.
Further, we propose to calculate histogram of the L channel of the image in CIELab color space and normalize it. Considering aforementioned assumptions, pixels of road marking lines are located in the upper right part of the distribution. For the histogram equalization the max-RGB like method is used [18]. This approach utilize the fact that humans perceive color relative to the contrast of the full image, i.e. difference between brightest and darkest point of the image. As a result, the histogram is normalized in such a way that its low boundary is a minimum value of a non-zero value of a brightness of the image in L channel, and upper boundary is maximum value of a brightness of the image in L channel.
Pixels of road marking lines on the image are located at the rightmost side of the normalized histogram (5% of the whole distribution). In order to choose appropriate threshold value the property of a Gaussian normal distribution and 3σ rule is used [19]. We calculate the mean μ and standard deviation σ for the normalized histogram of the image and use Gaussian distribution to estimate the threshold value. This value bounds 5% of the distribution of the brightest pixels which represent road marking lines. Example of approximation of a normalized histogram can be seen at Fig. 3, where red line represents normal distribution. The following equation is used for threshold value estimation: t = μ + σ(k + σ 2σu ), where t — threshold value; μ — mean value from normalized histogram; σ — standard deviation from normalized histogram; σu — standard deviation of an uniform distribution; k — scaling coefficient. (1) Histogram
Normalized Histogram
From (1) it is clear that the threshold value lies in the
half-interval (2σ; 3σ] and the exact value depends on standard
deviation. The standard deviation of the uniform distribution
is used to normalize the standard deviation of the histogram
and to accurately determine where the threshold value lies
between the mentioned interval. The scaling parameter k is
chosen depending on how much area of an image is covered
by road marking lines. For real road application we propose
to use k = 2. To illustrate the interval in which the threshold
value lies the image from Duckietown Project is used (Fig. 4)
[
If a scene of image is well-illuminated, its contrast is high . Thus, values on the histogram that correspond to the pixels of the road will be located near each other at the left side of the distribution, while values with intensities of a road marking lines will be located at the most right part of the histogram. In that case standard distribution will be rather small and the ratio of σ/σu consequently also will be small. Because of this threshold value will be close to 2σ, which guarantees that it will be less than values of the pixels of bright road marking lines. Contrariwise, when the scene of image is dark, histogram might not have distinct groups of pixels of the road or road marking lines. The standard deviation of such distribution will be greater than in the case described before, so the ratio of σσu will be relatively large. As a result, calculated threshold value will be close to 3σ, so it might be grater than values of the road marking lines, but it reduces amount of fake pixels marked as road marking lines (pixels of background or road itself).
The results of image binarization using described algorithm can be seen in Fig. 5. As a method for comparison Otsu binarization was chosen because it is one of the most popular method of threshold value calculation. Our approach for threshold value calculation shows better results: binary mask contains less noise and registers even yellow lines of road marking. This is possible because of those two assumptions at the beginning of this subsection.
After the initial processing stage (binarization) it is necessary to obtain information from the black-and-white bina ry image about which pixels belong to road marking and which are just road or other miscellaneous noise or even error from binarization algorithm used before. The proposed algorithm is as follows.
Given that we have the black and white binary mask and the original color image, we combine them into one resulting image. Pixels that were previously white receive color from the original image, and pixels that were plain black remain black.
This procedure is aimed to improve the next clustering part. This increases the useful information for the algorithm, because we will have not only information about the intensity of the pixels, but also the color, and therefore, it is easier to divide the points into meaningful clusters. For example, we can divide into separate groups the road lane and asphalt, which surrounds lane on all sides but has a completely different RGB color.
In this paper we selected HDBSCAN [
The algorithm has much fewer parameters that need to be configured, because during data processing HDBSCAN selects the best parameters according to its own indicators, for example, the epsilon parameter.
The algorithm can find clusters with densities varying within the cluster area.
The clustering procedure not only assigns a cluster number to each input point, but also calculates vector of probabilities where each probability reflects how probable a point belongs to each cluster or noise.
Now it is worth to emphasize why the main advantages of the clustering algorithm are critically important in lane finding scenario. Even on one route a car might encounter different light conditions, from completely dim (e.g tunnel) to sunny and bright, therefore, there is a need to develop a robust pipeline with several hyper-parameters to configure.
The quality of road markings in reality might be poor and the markings might be partially erased, therefore, it is necessary to be able to detect road markings with holes inside and at the same time to detect them as single cluster.
In the task of autonomous driving the safety comes first, so we select only the points with the lowest probability of noise. The result of this step of the pipeline is showed at Fig. 6.
IV.
The Fig. 7 represents visual part of comparison of the clustering algorithms. The parameters for clustering algorithms were chosen as follows: ◦ ◦ ◦ ◦ ◦ ◦
n clusters = 6, random state = 0,
eps = 10, min samples = 200,
min cluster size = 500, min samples = 200.
Interpretation of the visual result is as follows. The KMeans algorithm completely misses the actual clusters of the road and segments them only by y-value (Fig. 8b).
The result of DBSCAN is actually not so poor, since it clustered almost all the lanes into separate groups, with the exception of two lanes, which are located extremely right and left (Fig. 8c).
HDBSCAN showed better results than other algorithms, finding all the lanes on the image, although he found only part of the lane on the right (Fig. 8d). It is clear that further we can fit this points with the spline or parabola and get the actual lane.
Implementations of algorithms in the Python language were chosen. For HDBSCAN we used native authors’ implementations, DBSCAN and K-Means were used from scikit-learn library [23].
The bechmark was conducted on Raspberry Pi 4 model B with 64-bit ARM Cortex A72 CPU @ 1.5GHz and 4 GB of RAM. The procedure of the experiment was the following. The video with road was loaded and every clustering algorithm was consecutively run on each frame. Frame had the shape (120, 1200, 3) — height of 120 px, width of 1200 px and 3 RGB colors. The full results of the comparison can be found in Table
In order to test mesurable accuracy the proper dataset for testing should have be chosen. We have chosen Unsupervised Llamas (The unsupervised labeled lane markers dataset) [24]. II and I.
The results showed that the second worst result in terms of performance was shown by HDBSCAN with 12.5 FPS (frames per second). However, it is worth noting that the implementations of the algorithms are taken from different libraries, and because of this, the comparison is not very fair.
By using downscaled image there is a significant trade-off between time and precision, which is going to be shown in following section. The dataset consists of over 100 000 annotated images, so we took only 536 images from various parts of the dataset in order to test the proposed algorithm.
Overall, we achieve presion of 49% and AUC of 36%. As can be seen from Fig. 8 proposed solution is sometimes segments white car as road(c) and some artifacts are still present(d). Nonetheless, most of the images are good results such as (a) or (b). Decrease in precision and AUC can be explained by two things: 1) 2)
Firstly, we must specify ROI(Region Of Interest) on our own and farther parts of the road are removed from ROI and therefore are not being detected, which is okay, because there is no need to detect road markings from 200 meters from a car. But this markings are marked in the dataset and so in testing phase they are marked as non-detected.
Secondly, algorithm is not finding lanes that are on the sides, because they are harder to detect and have low-color intensity.
Future work can be focused on eliminating bad results such as (c) or (d) in Fig.8 by designing post-processing filter ste ps such as curv-fitting and discarding curves, that are incline d more horizontally, than vertically. Especially this step will be effective in eliminating noise that is coming from white car detection, because curve points are grouped in a form of a car, hence grouped more horizontally.
In order to understand how downscaling images beforehand affects the precision two tests were conducted. Test results with original size image and downscaled by factor of 0.3 is presented in III. As we can see, the decline of precision is not linearly proportional to decline of image size and this can be used to process images.
(a) Initial cropped road image (b) Result of K-Means (c) Result of DBSCAN (d) Result of HDBSCAN Fig. 7. Comparison between most popular clustering methods
In this article we have proposed a novel approach for detecting lanes on roads that is used for dim and low-light scenarios. Proposed method was successfully tested on the real road images taken from highway. The proposed method of clustering binarized images gave better results than others. As a topic for the future researches we would like to study possibility of using methods of illumination correction and histogram processing for more accurate calculating threshold value.
(a) Good result. Scale 0.3 (b) Good result. Scale 1 (c) Bad result. Car detection. Scale 0.3 (d) Bad result. Road artifacts. Scale 0.3