The visually impaired and blinds (VIBs) have made significant progress in social integration over the last five decades. This achievement is mostly based on inclusive smart technologies that create synergies between the community and VIBs [1]. Despite numerous assistive mobile applications (e.g., BDS (BeiDou Navigation Satellite System) WeChat and Google Maps) for spatial cognition, VIB navigation [2] remains problematic for the last mile, such as finding the entrance and identifying traffic signs [3][4][5][6][7][8][9][10][11].
In this study, a Java Android mobile application was developed to detect and recognize Kyrgyz traffic signs using the SIFT and BRISK (Scale-Invariant Feature Transform and Binary Robust Invariant Scalable Keypoints) methods [12,13] to localize keypoints and find sampling patterns with a two-keypoint binary descriptor, respectively. The true positive (i.e., recognition accuracy) and true negative (i.e., crucial mistakes) rates [14] are expected to be near 100 % at distances from 1.5 m to 3.5 m from the traffic sign.
To support VIBs, a new mobile application was developed to recognize traffic signs in Kyrgyzstan. Two key problems were solved in this study:
1. A novel technique has been applied for image processing. In the preprocessing step, the method π΅ππ‘πππ. πππππ‘ππππππππ΅ππ‘πππ creates a new bitmap scaled to a maximum resolution of 500 pixels with bilinear filtering. From up to four hundred keypoints detected by the SIFT method, two keypoints are selected. Then, the BRISK binary two-keypoint descriptor is designed employing a 291-point shape. The Hamming distance threshold is 19600 after five AdaBoost weak classifiers, which shows a true positive rate close to 100 % (smartphone Blackview BV6600 Pro -100 %, Doogee S96 Pro -75 % at 3.5 m) and a true negative rate of 100 % at distances from 1.5 m to 3.5 m. 2. A multithreaded Java Android application was developed using the CameraX library [15]. The image, smartphone soft-/hardware, and number of keypoints have an effect on the execution time. In the experiment with the traffic sign "Crosswalk left", the smartphone Doogee S96 Pro takes 0.1 s to find the sampling pattern using the knowledge base with 86 elements.
The World Health Organization showed that over 2.2 billion people experience vision impairment worldwide in 2021 [10], and hence assistive tools are in continuous demand. Traffic-sign recognition systems in cars [3] have been widely proposed for the market. The leading approach is based on convolutional neural networks and specific datasets, e.g., Tunisian traffic signs [4].
The percentage of wrongly recognized signs can reach 25 % [3], which is unacceptable for VIBs. Analysis of existing commercial products for VIBs, such as those referenced in [5][6][7][8][9][10][11], shows that they do not support the recognition of traffic signs related to pedestrians. Hence, the development of a mobile application that includes this functionality is a crucial task that should be undertaken to support the VIBs navigation near roads. Moreover, the usage of existing technologies is reasonable since it speeds up the development process, as was done in this study. The distance between the VIB and the traffic sign is assumed to be up to 4 m, which is the estimated width of the pedestrian path. The analyzed traffic signs are supposed to be of good quality and produced according to state standards. Google's Android platform has been taking over 70 % of the market share last five years. The CameraX Android API (application programming interface) is a Google Android native approach to work with different cameras on Android smartphones. CameraX is a Jetpack support library, which is considered the easiest way to make the Android camera application.
CameraX Android API Two-keypoint SIFT detector and BRISK descriptor with CameraX Android API employ the method presented in [16]. It and consists of three steps (see Figure 1):
1. Keypoints localization: The SIFT method is used to localize keypoints on the template image.
The BRISK method is employed to design image descriptors using pairs of keypoints that are empirically determined by an expert. This procedure will be automated in the future.
3. Image matching: Image capturing is performed using an Android CameraX library. The image is then downsampled with bilinear filtering in the method π΅ππ‘πππ. πππππ‘ππππππππ΅ππ‘πππ and converted to grayscale from RGB (Red-Green-Blue) format using the luminosity function [17]. The best pairs of keypoints calculated by the SIFT algorithm are selected for the design of the BRISK binary descriptor. Image matching uses AdaBoost classifiers and Hamming distance [18] (see Figure 2). Figure 3 presents two flowcharts: one for keypoints localization with the SIFT method and image descriptor design with the BRISK method (on the left), and another for the image matching algorithm (on the right).
As of August 2023, Kyrgyzstan had over 200 road signs [19], including 13 related to pedestrians (see Table 1).
The crosswalk signs "Crosswalk left", "Crosswalk right", and "Zebra crossing" are combined into a group "Crosswalk", as well as the signs "Emergency exit left/right" into "Emergency exit".
Regarding the SIFT keypoint localization, the AdaBoost cascade classifier is employed in this study. In this approach, scale-invariant locations of keypoints are searched in different scales. The convolution of two-dimensional Gaussian function G(x, y, ο³) and input grayscale image I(x, y) gives a filtered image:
where '*' is the convolution operation, ο³ is the population standard deviation, x and y are the pixel coordinates, and a Gaussian function:
In this study, the luminosity function [17] converts the image from RGB to greyscale I(x, y)
Unstable extrema are rejected if |D(π₯ )|<0.03. The extremum D(π₯ ) can be found by combining Eq. ( 6) and Eq. ( 5):
In this study, edges are detected employing a 2Γ2 Hessian matrix [12,20]:
where derivatives Dxx, Dxy, and Dyy are as follows:
(11) To eliminate the number of keypoints, the following inequality should be satisfied [12,16,20]:
In this study, the BRISK algorithm employs a binary descriptor with 291 points to depict the template image. The orientation and scale of the sample pattern are calculated using the positions of two keypoints. In the present software version, a human expert chooses two keypoints by examining keypoints with a consistent location at various octaves.
In the binary descriptor, 291 points are split as follows: 0-24 (1st group), 25-82 (2nd group), and 83-290 (3rd group). Figure 4 shows an example of the descriptor for the traffic sign "Above ground pedestrian crossing" (greyscale representation): (A) -points 0-24, (B) -25-82, (C) -83-290, (D) -0-290. The distance between any two points is computed via the Euclidean distance between two keypoints: for the 1st group point on the line connecting two keypoints, the distance to any nearest point is one-third of the distance between two keypoints. Each point n is associated with the mean pixel intensity I(n) in a circle of a radius 1/24, 1/16, or 1/12 of the Euclidean distance πΈ between two keypoints n1 and n2 [16].
, which is based on the comparison of average pixel intensities I(n1) and I(n2) at points n1 and n2, respectively:
where the string π·π is as follows: π·π = πΌ(π ) β πΌ(π ). (14) For a specific point n1, absolute differences πππ π·π are descending sorted within the appropriate group:
1. Points 0-24: the image binary descriptor considers the first 12 absolute differences for each point.
2. Points 25-82: the image binary descriptor considers the first 32 absolute differences for each point.
3. Points 83-290: the image binary descriptor considers the first 100 absolute differences for each point. Therefore, a total of 22956 absolute differences πππ π·π are considered in the image binary descriptor, which can be calculated as follows:
25 * 12 + 58 * 32 + 208 * 100 = 300 + 1856 + 20800 .
The knowledge base is stored on the smartphone internal storage and includes the following files [16]: 'root.txt'; N descriptor files "dN.txt"; "Description.txt"; N audio files "NameN.mp3". The knowledge base includes 86 sampling patterns (N=86; see Table 1) and has a size of 44.1 MB on the internal storage (106 MB in Random Access Memory (RAM) along with other data and code of the Android application), which is available on any up-to-date Android smartphone with operating system (OS) version 10 (10th and 11th are discussed in this study) or newer. The execution time for processing a test image (taken at sunny weather on a campus in Naryn, University of Central Asia, Kyrgyzstan; see Figure 5) on a Doogee S96 Pro smartphone is approximately 0.1 s.
In this study, the Hamming distance measures the similarity between two binary strings π΅π . The image-matching process employs five AdaBoost weak classifiers [21] that use binary decision trees.
To minimize the execution time of Java Android application, the maximum number of keypoints and the image dimension size are 400 and 500, respectively, which is compatible with any modern camera smartphone since a 2-megapixel sensor captures images of 1600ο΄1200 pixels. To avoid optical distortion effects [22] and locate the binary descriptor within the borders, a new image is created by adding 50-pixel margins to the original image. The method π΅ππ‘πππ. πππππ‘ππππππππ΅ππ‘πππ is used to downsample the original image with bilinear filtering. Then, a greyscale representation is calculated in eight parallel computational threads. To find keypoints, the image with a maximum dimension size of 500 pixels is smoothed five times. This process generates five scales using three groups (groups are applied sequentially until the target object is detected or not found) of the square matrix orders and population standard deviations in the Gaussian blur operator:
1. r=7, ο³=1; r=11, ο³=1.414214; r=13, ο³=2; r=19, ο³=2.828427; ο³=4. 2. r=11, ο³=1.414214; r=13, ο³=2; r=19, ο³=2.828427; r=25, ο³=4; r=35, ο³=5.656854. 3. r=5, ο³=0.5; r=7, ο³=0.70711; r=9, ο³=1; r=11, ο³=1.414214; r=13, ο³=2.
To reduce the execution time, four DoG images are calculated employing Eq. ( 4) in four parallel computational threads.
Keypoints are identified in two parallel computational threads for the third/second/first and fourth/third/second scales. Only 400 keypoints, which are closest to the center of the image according to the Euclidean distance, are considered. Figure 6 presents the scheme of how points are analyzed -the number of side points at the current level is two pixels larger than at the previous one: 1, 3, 5, 7, 9, etc.
The AdaBoost classifier is applied after discarding keypoint pairs whose descriptor points fall beyond the image boundaries:
where ft(BS) is AdaBoost weak classifier and T=5 [16]. If the Hamming distance surpasses the threshold 19600, the descriptor is of the template class. The threshold value was selected empirically based on the sum of the outcomes of the five weak classifiers mentioned above. Table 2 summarizes the speedup techniques used in this study.
In this study, a Java Android application "TrafficSignsKyrgyzstanWeCanSee" employs the proposed method to detect Kyrgyz traffic signs, and hence to support the spatial cognition and mobility of VIBs. Figure 7 Two smartphones, Doogee S96 Pro and Blackview BV6600 Pro, were used in the experiment. The true positive rate was calculated at different distances from 1.5 m to 5 m in three attempts. The results are shown in Figure 8. Figure 5 presents the photo taken during this experiment. The true positive rate was close to the required 100 % from a distance of 1.5 m to 3.5 m for the traffic sign "Crosswalk left": it was 100 % for the smartphone Blackview BV6600 Pro and 75 % for the smartphone Doogee S96 Pro at a distance of 3.5 m. The presented two-keypoint SIFT detector and BRISK descriptor showed that the true negative rate was 100 %.
In this study, a two-keypoint SIFT detector and a BRISK descriptor with CameraX Android API compose the approach to support the navigation and mobility of VIB pedestrians in Kyrgyzstan. In this method, keypoints are localized via the SIFT algorithm, and then selected pairs of keypoints are employed to design the sample pattern, i.e., the binary BRISK descriptor. The image matching is based on the Hamming distance and AdaBoost cascade classifier. The real-life experiment showed test results: a true negative rate is 100 % (this is a crucial parameter for VIBs) and a true positive rate is close to 100 %. In general, the traffic-sign recognition system satisfies requirements and hence can be implemented in practice. However, some elements (e.g., square matrix orders and population standard deviations in the Gaussian blur operator) of the presented approach are empirical, and therefore discussable.
In this study, a crucial VIB-assistive software, Java Android mobile application, was developed to recognize Kyrgyz traffic signs using two-keypoint SIFT detector, BRISK descriptor, CameraX Android API, and mp3 audio files to support the spatial cognition of VIBs near roads.
With a knowledge base of 86 sampling patterns, the mobile application shows the real-time performance: the execution time is 0.1 s for example with the traffic sign "Crosswalk left" (location is a campus in Naryn, University of Central Asia, Kyrgyzstan). In experiments on the distance from 1.5 m to 3.5 m, the presented SIFT/BRISK detector with two-keypoint descriptor showed 100 % true negative rate and a true positive rate close to 100 %: smartphone Blackview BV6600 Pro -100 %, Doogee S96 Pro -75 % on the distance 3.5 m.
The real-time performance is achieved using five AdaBoost classifiers for image matching and parallel computational threads for the greyscale representation of the color image (eight threads), calculation of DoG images (four threads), and localization of keypoints (two threads). Analysis of minimum requirements to the hardware shows that the mobile application is runnable on any up-to-date Android camera smartphone because it requires only 44.1 MB on the internal storage (106 MB in RAM along with other data and code) and a 2-megapixel sensor.
The most likely prospect for further development of this study is the design of an image descriptor that is geometrically close to the traffic signs.