The traffic-sign recognition system developed in this study aims to assist the spatial cognition and mobility of visually impaired pedestrians in Kyrgyzstan. The system employs a two-keypoint binary descriptor that implements the BRISK algorithm to find sampling patterns on the image. Pairs of keypoints are localized using the SIFT method. The developed Java Android mobile application implements the SIFT and BRISK approaches in real-time on Android CameraX using AdaBoost classifiers and multithreading. With a knowledge base of 86 sampling patterns, the execution time is 0.1 s for an example with the traffic sign “Crosswalk left”. In experiments conducted at distances 1.5 m to 3.5 m in the city of Naryn, Kyrgyzstan, the presented SIFT/BRISK detector demonstrated a true negative of 100 % and a true positive close to 100 % (Blackview BV6600 Pro and Doogee S96 Pro smartphones achieved 100 % and 75 %, respectively) rates at 3.5 m. This pilot project is expected to continue with more precise image descriptors for longer distances.
designed employing a 291-point shape. The Hamming distance threshold is 19600 after five AdaBoost weak classifiers, which shows a true positive rate close to 100 % (smartphone Blackview BV6600 Pro – 100 %, Doogee S96 Pro – 75 % at 3.5 m) and a true negative rate of 100 % at distances from 1.5 m to 3.5 m. 2. A multithreaded Java Android application was developed using the CameraX library [15]. The image, smartphone soft-/hardware, and number of keypoints have an effect on the execution time. In the experiment with the traffic sign “Crosswalk left”, the smartphone Doogee S96 Pro takes 0.1 s to find the sampling pattern using the knowledge base with 86 elements.
The World Health Organization showed that over 2.2 billion people experience vision impairment worldwide in 2021 [10], and hence assistive tools are in continuous demand. Traffic-sign recognition systems in cars [3] have been widely proposed for the market. The leading approach is based on convolutional neural networks and specific datasets, e.g., Tunisian traffic signs [4]. The percentage of wrongly recognized signs can reach 25 % [3], which is unacceptable for VIBs. Analysis of existing commercial products for VIBs, such as those referenced in [5-11], shows that they do not support the recognition of traffic signs related to pedestrians. Hence, the development of a mobile application that includes this functionality is a crucial task that should be undertaken to support the VIBs navigation near roads. Moreover, the usage of existing technologies is reasonable since it speeds up the development process, as was done in this study. The distance between the VIB and the traffic sign is assumed to be up to 4 m, which is the estimated width of the pedestrian path. The analyzed traffic signs are supposed to be of good quality and produced according to state standards.
Google’s Android platform has been taking over 70 % of the market share last five years. The CameraX Android API (application programming interface) is a Google Android native approach to work with different cameras on Android smartphones. CameraX is a Jetpack support library, which is considered the easiest way to make the Android camera application.
Two-keypoint SIFT detector and BRISK descriptor with CameraX Android API employ the method presented in [16]. It and consists of three steps (see Figure 1): 1. Keypoints localization: The SIFT method is used to localize keypoints on the template image. 2. Image descriptor design: The BRISK method is employed to design image descriptors using pairs of keypoints that are empirically determined by an expert. This procedure will be automated in the future. 3. Image matching: Image capturing is performed using an Android CameraX library.
The image is then downsampled with bilinear filtering in the method . and converted to grayscale from RGB (Red-Green-Blue) format using the luminosity function [17]. The best pairs of keypoints calculated by the SIFT algorithm are selected for the design of the BRISK binary descriptor. Image matching uses AdaBoost classifiers and Hamming distance [18] (see Figure 2). Figure 3 presents two flowcharts: one for keypoints localization with the SIFT method and image descriptor design with the BRISK method (on the left), and another for the image matching algorithm (on the right).
As of August 2023, Kyrgyzstan had over 200 road signs [19], including 13 related to pedestrians (see Table 1).
The crosswalk signs “Crosswalk left”, “Crosswalk right”, and “Zebra crossing” are combined into a group “Crosswalk”, as well as the signs “Emergency exit left/right” into “Emergency exit”.
Regarding the SIFT keypoint localization, the AdaBoost cascade classifier is employed in this study. In this approach, scale-invariant locations of keypoints are searched in different scales. The convolution of two-dimensional Gaussian function G(x, y, ) and input grayscale image I(x, y) gives a filtered image:
(
1 2 . ) ∗ 0.21 + . , and (
. ) ∗ 0.007, ⁄ . (
) ∗ 0.72 + are Java methods, is the smallest using Android class
( , ) = where . , : . .
( + ( , , ) = ( , , ) − ( , , ). element that can be addressed in a raster RGB image.
The DoG (Difference of Gaussians) function ( , , ) is employed to find keypoints that are stable across different scales. DoG is the result of subtracting two neighbour scales that are smoothed by Gaussian filters with a different weight k:
The DoG function is the approximation of the scale-normalized Laplacian of Gaussian 22G. Building DoG ( , , ) follows the method proposed in [16]. To generate five SIFT scales, four images (maximum dimension sizes are 180, 340, 680, and 1360 pixels, i.e., four SIFT octaves) are smoothed five times in the Gaussian blur operator with five square matrix orders r and population standard deviations :
180: r=5, =0.707107; r=7, =1; r=11, =1.414214; r=13, =2; r=19, =2.828427. descriptor design with the BRISK method (left flowchart) and the image matching algorithm (right flowchart)
where ‘*’ is the convolution operation, is the population standard deviation, x and y are the pixel coordinates, and a Gaussian function:
In this study, the luminosity function [17] converts the image from RGB to greyscale I(x, y) 2. 340: r=11, =1.414214; r=13, =2; r=19, =2.828427; r=25, =4; r=35, =5.656854. 3. 680: r=19, =2.828427; r=25, =4; r=35, =5.656854; r=49, =8; r=69, =11.313708. 4. 1360: r=35, =5.636854; r=49, =8; r=69, =11.313708; r=97, =16; r=137, =22.627417.
Detection of keypoint candidates (i.e., local maxima and minima of DoG function) is similar to the presented in [16, 20] methodology. Keypoints are rejected using the Taylor expansion of (, , ):
1 (
2 where x=(, , ) is the offset from and D and its derivatives are computed at a particular point. To find the extremum , the equation ′ ( ) = 0 should be solved (′ ( ) is the derivative of D with respect to x):
In this study, the BRISK algorithm employs a binary descriptor with 291 points to depict the template image. The orientation and scale of the sample pattern are calculated using the positions of two keypoints. In the present software version, a human expert chooses two keypoints by examining keypoints with a consistent location at various octaves.
In the binary descriptor, 291 points are split as follows: 0-24 (1st group), 25-82 (2nd group), and 83-290 (3rd group). Figure 4 shows an example of the descriptor for the traffic sign “Above ground pedestrian crossing” (greyscale representation): (A) – points 0-24, (B) – 25-82, (C) – 83290, (D) – 0-290. The distance between any two points is computed via the Euclidean distance between two keypoints: for the 1st group point on the line connecting two keypoints, the distance to any nearest point is one-third of the distance between two keypoints. Each point n is associated with the mean pixel intensity I(n) in a circle of a radius 1/24, 1/16, or 1/12 of the Euclidean distance
between two keypoints n1 and n2 [16].
The Hamming distance is calculated via the binary string , which is based on the comparison of average pixel intensities I(n1) and I(n2) at points n1 and n2, respectively: , .
,
Therefore, a total of 22956 absolute differences binary descriptor, which can be calculated as follows: are considered in the image 25 * 12 + 58 * 32 + 208 * 100 = 300 + 1856 + 20800 . (7) (8) (9) (10) (11) (12) (13) (14) where the string
is as follows: For a specific point n1, absolute differences are descending sorted within the Points 0-24: the image binary descriptor considers the first 12 absolute differences for Points 25-82: the image binary descriptor considers the first 32 absolute differences for Points 83-290: the image binary descriptor considers the first 100 absolute differences appropriate group: 1.
for each point.
The knowledge base is stored on the smartphone internal storage and includes the following files [16]: ‘root.txt’; N descriptor files “dN.txt”; “Description.txt”; N audio files “NameN.mp3”.
The knowledge base includes 86 sampling patterns (N=86; see Table 1) and has a size of 44.1 MB on the internal storage (106 MB in Random Access Memory (RAM) along with other data and code of the Android application), which is available on any up-to-date Android smartphone with operating system (OS) version 10 (10th and 11th are discussed in this study) or newer. The execution time for processing a test image (taken at sunny weather on a campus in Naryn, University of Central Asia, Kyrgyzstan; see Figure 5) on a Doogee S96 Pro smartphone is approximately 0.1 s.
In this study, the Hamming distance measures the similarity between two binary strings . The image-matching process employs five AdaBoost weak classifiers [21] that use binary decision trees.
To minimize the execution time of Java Android application, the maximum number of keypoints and the image dimension size are 400 and 500, respectively, which is compatible with any modern camera smartphone since a 2-megapixel sensor captures images of 16001200 pixels. To avoid optical distortion effects [22] and locate the binary descriptor within the borders, a new image is created by adding 50-pixel margins to the original image. The method . is used to downsample the original image with bilinear filtering. Then, a greyscale representation is calculated in eight parallel computational threads.
To find keypoints, the image with a maximum dimension size of 500 pixels is smoothed five times. This process generates five scales using three groups (groups are applied sequentially until the target object is detected or not found) of the square matrix orders and population standard deviations in the Gaussian blur operator: 1. r=7, =1; r=11, =1.414214; r=13, =2; r=19, =2.828427; r=25, =4. 2. r=11, =1.414214; r=13, =2; r=19, =2.828427; r=25, =4; r=35, =5.656854. 3. r=5, =0.5; r=7, =0.70711; r=9, =1; r=11, =1.414214; r=13, =2.
To reduce the execution time, four DoG images are calculated employing Eq. (
Keypoints are identified in two parallel computational threads for the third/second/first and fourth/third/second scales. Only 400 keypoints, which are closest to the center of the image according to the Euclidean distance, are considered. Figure 6 presents the scheme of how points are analyzed – the number of side points at the current level is two pixels larger than at the previous one: 1, 3, 5, 7, 9, etc.
The AdaBoost classifier is applied after discarding keypoint pairs whose descriptor points fall beyond the image boundaries: (15) ( ) =
( ), where ft(BS) is AdaBoost weak classifier and T=5 [16]. If the Hamming distance surpasses the threshold 19600, the descriptor is of the template class. The threshold value was selected empirically based on the sum of the outcomes of the five weak classifiers mentioned above. Table 2 summarizes the speedup techniques used in this study.
Speedup technique Eight parallel computational threads Four parallel computational threads Two parallel computational threads Five AdaBoost classifiers In this study, a Java Android application “TrafficSignsKyrgyzstanWeCanSee” employs the proposed method to detect Kyrgyz traffic signs, and hence to support the spatial cognition and mobility of VIBs. Figure 7 presents the screenshot, an original image taken by the smartphone Doogee S96 Pro at cloudy weather (location is a campus in Naryn, University of Central Asia, Kyrgyzstan), and a greyscale image with keypoints (fuchsia and turquoise colors are used for third/second/first and fourth/third/second DoG functions, respectively). Two other Java Android 10 applications (approximately 72 % of Android smartphones can run these applications as of August 2023) were designed in Android Studio 4.0: 1. Localization of all keypoints using SIFT method. 2. Identification of keypoints’ pairs and design of the sample pattern via the BRISK descriptor.
Two smartphones, Doogee S96 Pro and Blackview BV6600 Pro, were used in the experiment. The true positive rate was calculated at different distances from 1.5 m to 5 m in three attempts. The results are shown in Figure 8. Figure 5 presents the photo taken during this experiment. The true positive rate was close to the required 100 % from a distance of 1.5 m to 3.5 m for the traffic sign “Crosswalk left”: it was 100 % for the smartphone Blackview BV6600 Pro and 75 % for the smartphone Doogee S96 Pro at a distance of 3.5 m. The presented two-keypoint SIFT detector and BRISK descriptor showed that the true negative rate was 100 %.
In this study, a two-keypoint SIFT detector and a BRISK descriptor with CameraX Android API compose the approach to support the navigation and mobility of VIB pedestrians in Kyrgyzstan. In this method, keypoints are localized via the SIFT algorithm, and then selected pairs of keypoints are employed to design the sample pattern, i.e., the binary BRISK descriptor. The image matching is based on the Hamming distance and AdaBoost cascade classifier. The real-life experiment showed test results: a true negative rate is 100 % (this is a crucial parameter for VIBs) and a true positive rate is close to 100 %. In general, the traffic-sign recognition system satisfies requirements and hence can be implemented in practice. However, some elements (e.g., square matrix orders and population standard deviations in the Gaussian blur operator) of the presented approach are empirical, and therefore discussable.
In this study, a crucial VIB-assistive software, Java Android mobile application, was developed to recognize Kyrgyz traffic signs using two-keypoint SIFT detector, BRISK descriptor, CameraX Android API, and mp3 audio files to support the spatial cognition of VIBs near roads.
With a knowledge base of 86 sampling patterns, the mobile application shows the real-time performance: the execution time is 0.1 s for example with the traffic sign “Crosswalk left” (location is a campus in Naryn, University of Central Asia, Kyrgyzstan). In experiments on the distance from 1.5 m to 3.5 m, the presented SIFT/BRISK detector with two-keypoint descriptor showed 100 % true negative rate and a true positive rate close to 100 %: smartphone Blackview BV6600 Pro – 100 %, Doogee S96 Pro – 75 % on the distance 3.5 m.
The real-time performance is achieved using five AdaBoost classifiers for image matching and parallel computational threads for the greyscale representation of the color image (eight threads), calculation of DoG images (four threads), and localization of keypoints (two threads). 100 % ,te 80 a re 60 v tii 40 s po 20 e ru 0 T 1.5 2
Analysis of minimum requirements to the hardware shows that the mobile application is runnable on any up-to-date Android camera smartphone because it requires only 44.1 MB on the internal storage (106 MB in RAM along with other data and code) and a 2-megapixel sensor.
The most likely prospect for further development of this study is the design of an image descriptor that is geometrically close to the traffic signs.