-In this paper, an application of tools for global image analysis, provided by the geometrized histogram method for detecting and recognizing basic elements of road scenes, is described. The application to finding permanent and temporary road markings, detecting stop lines and features showing the fact of approaching to crossings are addressed. The possibility of using the technique in detecting and recognizing traffic signs and traffic lights is discussed. The approach is compared with other known methods for solving the described tasks. The algorithms for solving the considered tasks and their program implementation are outlined. The results of operation of the developed software system for particular images and video sequences are presented. The problems of designing a software system for integrated understanding road scenes, involving analyzing the road, roadsides, and other objects on the road, the sky, road markings, and traffic signs, are discussed.
In this paper, a new approach is proposed for detecting and
recognizing basic informative elements of road scenes, such
as road marking and traffic signs, signals of traffic lights, etc.
This topic is very urgent, and the application of the obtained
results can be important in designing Advanced
DriverAssistance Systems (ADAS), widely implemented by the
world leading automobile manufactures. This topic is also
important for designing control systems for driverless
vehicles. Unfortunately, the existence of unsolved problems
in this field is proved by accidents with driverless vehicles
(frequently, fatal). The solution of these problems is
especially topical for the countries with the road and climate
conditions similar to those in the Russian Federation. One of
the most recent surveys of papers and results obtained in the
field of detecting and recognizing road marking can be found
in [
The first publication of the authors on this topic can be
found in [
The problems of detecting stop lines are also discussed. Algorithms for finding traffic signs (mainly connected with regulation of motion on crossings) are presented. The technique for finding traffic lights is also described. In Section 2, the geometrized histograms method for concise image description and segmentation is briefly described. Its tools for global image analysis are presented. In Section 3, these tools are applied to the problem of detecting and recognizing road marking. Algorithms for finding stop lines, traffic signs, and traffic lights are outlined in Section 4.
In contrast to the main methods for image segmentation, the
geometrized histograms method is designed so that the main
processing of video data can be made in parallel. Using the
technique described in [
The geometrized histogram describes rather exactly the
value distribution of the function specifying the image in its
rectangle. The geometrized histogram is obtained by
projecting pixels of the strip on its lower side. For a
grayscale image [
axis Os.
The range HSg = [HminSg, HmaxSg] and the mean value of the hue Hmean Sg.
The range SSg = [SminSg, SmaxSg] and the mean value of the saturation SmeanSg.
The range ISg = [IminSg, ImaxSg] and the mean value of the intensity ImeanSg.
approximately to the number of points of the strip located in the strip over Sg and having the color characteristics within the boundaries specified above for it).
Denote by dens(Sg) = CardSg/( endSg endSg + 1 ) the
density of Sg. It is the substantial property of the algorithm
of obtaining the geometrized histogram in a strip that it can
be obtained for one pass of the array of the image pixels in
the strip. It makes it possible to obtain it in real time. The
union of the geometrized histograms of all strips give the
geometrized histogram of the image. Using the original
clustering operations [
On the set of color bunches STG a “search lattice”
SearchLat is constructed [
B. Construction of global objects using the geometrized histograms method
We mean by global objects subsets of color bunches
containing color bunches located in several adjacent strips.
Global objects are constructed from left and right germs of
global objects (left and right contrast curves) [
When we deal with chains of color bunches representing road markings, we should keep in mind that because of inhomogeneous illumination and shadows, the intensity and color characteristics of the corresponding color bunches may of bunches adjacent in the strip may vary significantly. In the next subsection, the principles for constructing chains of color bunches that can represent a road marking will be described.
C. Principles of construction of objects in STG that are
candidates for the image in STG of a road marking
Methods for constructing candidates for an image of a
road marking in STG have been briefly outlined in [
If we deal with the problem of finding rather “narrow”
objects, such as road markings, chains of color bunches,
which belong to adjacent strips, contain both dominating and
dominated color bunches [
Figure 1 presents an example of an image with the detected road marking (the right part), the grayscale image corresponding to the color source image with superimposed color bunches (the middle part), and the candidates for being the images of parts of road marking in strips (the right part). Candidates for the image of parts of road marking in STG in strips were obtained using the search lattice SearchLat and the method (reasoning system) described above. Since there is no assumption that the vehicle is located on the road in the system, it is possible to find the marking on a square or in parking places. Since there are no data on the camera parameters and the position of the vehicle on the road, some candidates in the presented image look strange, but in constructing chains and recognizing true candidates, false candidates are eliminated (the right part of Fig. 1).
Let us describe the rules for generating chains of candidates for parts of marking in strips. These chains are called candidates for the image of road marking in STG. Chains are constructed from the zero (bottom) strip. All bunches involved in a certain candidate are eliminated from the finding next candidates. Assume that a chain of bunches bi i = 0, …, k, located in a chain of adjacent strips, has been already constructed. Denote by Inti = [begi, endi] the localization intervals of the bunches bi, by ei the vectors that join the ends of the interval Inti-1 and Inti, and by Di the vectors of directions joining the ends of Int0 and Inti. In constructing continuous left chains of intervals, we deal with the left ends of intervals, while dealing with the right chains– with the right ones. Since there is no assumption on the position of the vehicle on the road, at the first step of the algorithm, we require only that the intervals Int0 and Int1 have a strong intersection. Let us introduce in the consideration the reduced direction vectors di = Di/(i - 1). The vectors ei and di have the x coordinate equal to the strip width, their direction vector is parameterized by the y coordinate. The continuity conditions are formulated in terms of relations between ei and di and restrictions on the jumps of the corresponding angles. Applying these rules, we obtain a set of candidates for the image of the road marking in STG. The matrices of traces of marking candidates on the sets of candidates for parts of the marking in strips are constructed. Matrices of traces show the number of the candidate for SearchLat, a reasoning system is designed, which compares the parameters of a certain color bunch with those of the adjacent basic bunches from SearchLat. This system makes a conclusion on whether this color bunch belongs to candidates for being an image of a part of road marking in the part of STG connected with the given strip. To determine the difference in intensity, only small thresholds are employed at the level of distinguishability of human vision system (of order of several intensity grades), which can take place between the road marking and road when the illuminations varies in a wide range. global marking (left and right separately) that passes in particular strips through each local candidate in each strip. Using the trace matrices, we can establish relations between left and right candidates for the global road marking. The related left and right candidates pass through the same candidates in strips. In the matter of fact, each real global candidate (left or right) has a small number of connected (right or left) candidates. The real road marking has to generate both left and right candidates, which pass through the same color bunches, i.e. they contain the same bunches in STG. In the ideal case, one left global candidate corresponds to one right global candidate. In the case of segmentation faults, the connected candidate can be divided into several connected candidates. For example, it may occur that in a number of strips, the marking disappears (occlusion) or because of noise, some false bunches connected with another object (a part of another vehicle or a puddle) may be added. III. ROAD MARKING RECOGNITION BASED ON CANDIDATES CONSTRUCTED IN STG AND COMPARISON WITH THE OTHER
METHODS
Among the constructed continuous chains of candidates
for road marking in strips, i.e. images of the marking in
strips, false chains may be found. These chains may be
determined by road railings, other participants of the traffic,
wet places, etc. The presence of such false objects is noted in
all main papers on this subject [
the internal and external camera parameters and operates well when the inclination of the road changes sharply, in contrast to the methods using the flat model of the road vicinity.
The method creates a big list of candidates for road marking parts, including possible false candidates such as, railings, parts of other participants of the traffic, posts, wet places, etc. Then there is a reasoning system that based on comparative logical analysis selects true parts of the road marking. In each processed frame, program gives a convenient description of the list of details of parts of marking, such as inclination, curvature, the distances between parts of marking, their intensity-color characteristics. These data can be quickly and efficiently used for estimating the next frame.
The method does not impose the restrictions on the shape of road marking and is efficient under sharp changes of the shape of temporary marking, in contrast to methods based on the Hough transform. It is also possible to find the marking of adjacent lanes and construct the marking on distant parts of the road. performed for candidates for white and yellow marking separately. It is explained by the fact that in the winter period white road marking may have a slightly yellow color, and it is a particular intelligent problem to distinguish both types of road marking during this period. Candidates for yellow road marking start from color bunches with intensity-color characteristics in the yellow-orange domain. As the initial bunches, the bunches with a rather big saturation are chosen. Then a fall of saturation is possible (which is really occurred in images shot by real cameras), and the construction is continued based mainly on the geometric fitting of localization intervals.
Figure 2 presents examples of the operation of the
proposed reasoning system in the form of several frames
from records of the results of processing a particular video
sequence. The complete records of processing this video
sequence can be found in [
(1280х720, 1920х1080), which is reached on standard personal computers.
Simultaneously, complete segmentation of the frame
is performed, and such important objects as the sky,
roadsides, the road, other participants of the traffic,
dangerous objects, like posts near the road are
detected [
The performance of developed method was studied using
video sequences of road scenes shot by different cameras,
even by cheap car video registrators. The detailed description
of the developed software system implementing the
geometrized histograms method and the image understanding
systems for the whole list of tasks was presented in [
The development of the reasoning system is under progress and some disadvantages are being eliminated.
IV. DETECTION AND ANALYSIS OF STOP LINES, TRAFFIC
LIGHTS, TRAFFIC SIGNS
These elements should be evaluated in producing a
control mode for an autonomous vehicle when going into a
crossing, as well as in order to warn the driver in ADASs.
Using even rough camera calibration of the camera close
zone, knowing the approximate distance from the vehicle to
upper boundaries of the closest strips of the image, it is easy
to detect the solid marking near the vehicle. The solid lane
marking is one of the features of the approaching stop line.
As processing of a big number of images has shown, a rather
distant stop line is represented in STG by one or several
dominated color bunches in a certain strip in front of the
vehicle. These bunches overlap the lane. The intensity-color
characteristics of these bunches are close to those of the lane
marking. These bunches are found by simple search in STG.
While approaching the stop line, these bunches become
dominating. A part of the image containing candidates for
the stop line is cut from the image. Using the division of the
cut image into vertical strips, the stop line is exactly
localized. To detect a real stop line, it is necessary to analyze
additional features (the red traffic light, stop traffic sign, give
way sign).An important component of planning the motion
on crossings is to detect traffic signs and to determine their
state. The geometrized histograms method makes it possible
to detect in images contrast color objects of size less than 3
pixels. To detect traffic lights in advance, it is necessary to
work with HD video in real time. At present, using
multithreading in constructing STG on standard processors
with four 4 CPU cores, the performance is about 50 fps for
resolution of 1280х720 (about 25 fps for 1920х1080) for 2.5
GHz CPUs [
The detection and recognition of traffic signs are other important elements of controlling the motion into crossings. The traffic signs (give way, stop, main road, etc.) contain informative color parts and even the combination of these components (e.g. green and blue ones in the traffic sign of pedestrian crossings). As a rule, these components have rather big saturation and can be effectively found in STG. After localizing a candidate, a small its neighborhood is cut and STG is constructed for it. Methods have been developed that allows one to find in STG typical geometrical figures (rectangles, rhombs, triangles, etc.). This specific technique will be published separately.
Complex scene analysis is a substantial element of scene analysis. The detection of road marking under the bridge (Fig. 2) makes it possible to save the correct motion direction, when the other systems (detection of the road, roadsides, and sky) produce wrong results. It is very important to have a reasoning system that takes into account the results for previous frames. At present this system is under development. A particular publication will be devoted to this topic.
In this paper, algorithms for detecting and recognizing road marking were briefly described. Examples of operation of a software system implementing these algorithms were presented. Estimates of the performance of this system were given for HD video. Algorithms for solving tasks for finding stop lines, traffic lights, and traffic signs based on the graph STG were also outlined.
This work was supported by the Russian Foundation for Basic Research, projects N. 18-07-00127 and 19-08-01159.