In paper proposed approach for boost the inclusion and safety of visually impaired people by creating method of augmented reality audiostream creation which contains objects and named persons identified and classified from a video stream using CNN. Developed method differs from existing ones by two advantages: ability to classify named individuals in the video stream for boosting the inclusion of visually impaired people; and ability to consider of prioritizes dangerous objects when generating audiostream for boosting the safety of visually impaired people. By this, method is contributed to achieving UNDP SDG No. 3, No. 10 and No. 11. For research the method effectiveness, test software was developed. Obtained the applied comparison results of neural network models YOLOv5n, YOLOv5x, YOLOv8n, YOLOv8x, YOLOv11n and YOLOv11x revealed that the YOLOv11x model is better suited for objects identifying and classification in environment; obtained sufficient values of macrometrics for 80 objects classes: Accuracy 74.1%, Precision 73.74%, Recall 65.85%, mAP@0.5 71.34%, and mAP@0.5:0.95 54.85%. Obtained the applied results from CNN neural network model macrometrics indicate high-quality classification of named persons: Accuracy 97%, Precision 95%, Recall 96%, and F1-score 95%.
According to the World Health Organization, there are about 2.2 billion people with visual
impairments in the world, and their number is increasing every year due to population aging and
insufficient access to health services in developing countries [
There are also challenges with the use of current technologies. For example, while mobile
applications based on artificial intelligence help
with navigation, they often have limited
functionality or low accuracy [
Other problems include psychological barriers [
The use of such technologies will contribute to increasing inclusion and safety for people with
visual impairments and creating an environment that takes into account the needs of people with
disabilities, and the created software will help achieve the UN Sustainable Development Goals
(SDGs), in particular SDG No. 3 «Promote healthy lives and well-being for all ages», SDG No. 10
«Reduce inequalities within and among countries» and SDG No. 11 «Ensure accessible, safe,
sustainable and environmentally sustainable cities and human settlements» [
Research goal is to boost the inclusion and safety of visually impaired people by creating an audio stream of objects and named persons identified and classified from a video stream using deep convolutional neural networks.
The main paper contribution is the proposed method of augmented reality audiostream creation using CNN, which differs from existing ones in that when classifying objects in a video stream, it is able to recognize named persons among them, which contributes to increasing the level of inclusion. Also, thanks to the introduced system of listing priority objects by danger, the proposed method can be used to increase the safety of people with visual impairments.
The relevance of using information technology to improve inclusion and safety for people with
visual impairments is undeniable. Some researchers focus on integrating multisensory data,
combining images, sounds, and information from different sensors, creating a more complete
picture of the environment [
A walking assistance system for people with visual impairments using XR glasses for safe
movement on the street is proposed in the study [
The authors of study [
In [
A voice application for smartphones was created in [
In paper [
The authors of the article [
In [
In the paper [
The authors of the study [
A method combining LBPH and CNN for image preprocessing using equalized histograms was
proposed in the study [
Analysis of recent studies allows us to conclude that there are unresolved problems on the way to increasing the level of inclusion. In particular, the identification of named persons, which can promote social communication and a sense of security, has not been implemented. Meanwhile, existing solutions do not take into account priorities for identified environmental objects, although the found objects in different situations may have diametric priorities.
In this regard, the study aims to increase the level of inclusion by identifying named persons, and makes an attempt to increase the level of safety of people with visual impairments by introducing priorities for identified environmental objects.
Artificial intelligence tools and methods allow us to create systems to support people with special needs. One important direction is the creation of an augmented reality audio stream that can increase inclusion and safety for people with visual impairments by analyzing the environment through a video stream. Figure 1 proposes a general approach to converting input information in the form of a video stream and a prioritized list of dangerous objects for people with visual impairments into output information in the form of an audio stream.
In accordance with the presented approach in Figure 1, a diagram of the method of augmented reality audiostream creation using CNN is shown in Figure 2. As part of the conducted research, the use of YOLO models for identification and classification of objects from the video stream and CNN for classification of named individuals is proposed.
The input data of the method are a video stream, a neural network model for object identification, a priority list of dangerous objects for people with visual impairments, a delay in voicing objects, as well as a confidence threshold for identifying objects in the video stream and a neural network model for classifying named persons. The video stream provides dynamic visual information about the environment in which a person with visual impairments is located, and neural network models process this video stream to identify and classify objects according to a set confidence threshold from which an audio stream is formed, which is relevant for people with visual impairments.
The first stage of the method is the pre-processing of the video stream. Preprocessing for the
YOLO neural network model [
The second stage includes determining the boundaries of objects on the received and preprocessed in stage 1 frames of the video stream and estimating the probability of their belonging to a certain class. YOLO performs classification of objects in the frame by one-stage detection, dividing the image into an S×S grid. The grid size depends on the YOLO versions and the detection level and can be: 80×80, 40×40, 20×20 cells, where each cell is responsible for predicting bounding boxes and class probabilities for objects located within its boundaries. The neural network architecture simultaneously calculates the coordinates of the frames, the degree of confidence in the presence of the object and its class affiliation, using a forward pass. After that, confidence thresholds and the Non-Maximum Suppression algorithm are applied to filter out bounding boxes and leave only the most relevant ones. Using the confidence threshold set by the user (from 0.0 to 1.0), filtering of objects occurs: only those whose probability exceeds this threshold will be selected for output to the audio stream in the following stages. If an object is detected in the frame that is identified as a person with an additional threshold value above 0.6, an additional determination of this person is performed at stage 3.
In the third stage, the named persons are classified. First, the boundary of the identified person in the frame is determined, after which the probability with which this person belongs to a specific class is calculated. The class of the named person in the next stage will be announced as "person is"+name of person. If the person could not be identified, his name will not be announced in stage 4, but "unknown person" will be announced instead. Using YOLO to identify named persons is not advisable, since this approach is optimized for fast detection of object classes, and not for accurate recognition of specific persons. YOLO performs well in identifying people in the frame, but its final layers are not designed to extract unique features necessary for identifying and classifying named persons. For the identification and classification of named persons, a CNN neural network model is used in the study. The steps involved in face identification using a cascade classifier based on Haar functions and classification of a named person using it using CNN are shown in Figure 3.
To classify a person detected in a frame from a video stream, a cascade classifier based on Haar
functions is used to determine the boundaries of the face [
The fourth stage involves the formation of an audio stream. The classes of objects from the video stream are sorted for voiceover according to the defined priorities of danger for visually impaired people, as well as with identified named persons. It is important to note that if several objects with the same priority are classified in the frame, they will be voiced in the order in which YOLO classifies them. If the user has set a priority for the "person" class, then all persons (including named persons) will be voiced according to this priority. If there are several named persons in the frame, they will be voiced in the order of their classification. If the priority for the "person" class is not set, named persons will be voiced after all objects with a higher priority, in the order in which they were classified. The voiceover update frequency is set by the user to prevent excessive load on the perception of information. Since this study is experimental in nature, the audio stream is updated within a 5-second interval, which can be changed by the user if necessary, and the output of classified objects to the application log has no delays. As part of the study, the following list of priority objects for visually impaired people was determined: bicycle, car, motorcycle, bus, train, truck, traffic_light, stop_sign. The specified list is purely experimental and can be changed or supplemented if necessary.
The initial data of the method are classes of objects with an assessment of the confidence of their belonging to the corresponding classes, classes of classified named persons, if they have been identified, as well as an audio stream formed on the basis of the priority of dangerous objects for visually impaired people, and containing the specified classes of objects and named persons, if such are classified.
Thus, the presented method of augmented reality audiostream creation using CNN can contribute to increasing inclusion and safety for people with visual impairments.
To test the presented method, it is necessary to study the existing pre-trained YOLO neural network models, as well as to investigate the influence of the batch size and the number of epochs when training the CNN model on the accuracy of the classification of named persons by test software (Figure 4).
а) b)
In the process of experimental research, test software was created for creating an augmented
reality audio stream in Python (Figure 4), which allowed testing the proposed method of
augmented reality audiostream creation using CNN.To implement this software prototype, the
following Python libraries were used: «tensorflow» [
To study the models, the COCO dataset was used, which contains 330,000 images (of which over
200,000 are annotated), 80 object classes [
To study of object classification at stage 2 of developed method, the YOLOv5n, YOLOv5x, YOLOv8n, YOLOv8x, YOLOv11n, and YOLOv11x models were compared, which are pre-trained to classify 80 environmental objects, including, for example: car, motorcycle, traffic light, bench, dog, backpack, tennis racket, apple, chair, toothbrush, etc. The models were compared using the metrics Accuracy, Precision, Recall, etc. [44, 45]. Pre-trained objects can be divided into the following categories: people, transport, infrastructure, animals, personal belongings, sports equipment, dishes and kitchen items, food, furniture, household appliances, stationery. Figure 5 shows the results of the macrometrics Accuracy, Precision, Recall, mAP@0.5, mAP@0.5:0.95 for each of the tested YOLO models.
The study found that YOLOv11X has the best metrics for classifying objects in a video stream. In particular, Accuracy 74.1%, Precision 73.74% and Recall at 65.85% indicate that the model can correctly identify objects. The obtained mAP@0.5 indicator (average accuracy at the IoU threshold of 0.5) is 71.34%, and mAP@0.5:0.95 has a value of 54.85%. The obtained results of these indicators indicate that the model finds objects well even with a small error in positioning. In modern computer vision systems aimed at helping people with visual impairments, one of the key indicators is the speed of real-time image processing. In particular, the frame rate per second (FPS) determines how quickly the model can analyze the environment and provide the user with relevant information. High FPS is important for ensuring continuous and accurate object recognition, especially in changing environmental conditions. As part of the study, FPS was calculated for the YOLOv5x, YOLOv8x, YOLOv11x models, which allows us to assess their suitability for real-time application. The models were tested on a video [46] containing scenes with people and cars. The results of FPS calculations are presented in Table 1.
The YOLO model test results show a noticeable increase in FPS when moving to newer versions. In particular, the YOLOv5x model has the lowest average FPS (143.47), which means a longer frame processing time, while YOLOv8x improves this figure to 155.32 FPS, reducing the total processing time by 0.4 seconds. The best results are shown by YOLOv11x, which reaches 182.66 FPS, which is 27% faster than YOLOv5x and almost 18% faster than YOLOv8x.
The reduction in the total frame processing time from 5.23 seconds (YOLOv5x) to 4.11 seconds (YOLOv11x) indicates a significant increase in efficiency, which is critical for real-time object recognition, especially for systems aimed at assisting the visually impaired, where even small delays can affect safety and usability.
The FPS indicators given in Table 1 were calculated during the processing of the video stream by YOLO models without displaying the image on the screen and without visualizing the superimposed bounded boxes. At the same time, for the correct display of the video in the software prototype together with the visualization of the superimposed bounded boxes of the detected objects, the optimal frame rate of 30 FPS was empirically determined, which can be changed in the settings depending on the user's requirements and the computing resources of the system.
Considering the conducted studies, YOLOv11x is the best of the tested models, providing the highest indicators of Accuracy, Precision, Recall, mAP@0.5, mAP@0.5:0.95, FPS and the lowest total frame processing time, therefore, the micrometric indicators of the YOLOv11x model were further investigated in more detail. Figure 6 shows the micrometric indicators for each class of only the priority objects in terms of danger for people with visual impairments, which were listed in section 3.
The evaluation of the YOLOv11X model performance indicates its balanced performance in classifying objects of different classes. The Precision value is stable for most classes, approaching 73.74%, which indicates the overall high ability of the model to correctly classify objects without a significant number of false positives. However, the Recall indicator shows significant variability, which may indicate the heterogeneous ability of the model to find all objects of certain categories in the input images.
The lowest Recall level is observed for the «bicycle» class (36.06%), which indicates the difficulty of model in detecting this type of object due to the peculiarities of their shape, size, or the presence of partial overlap with other objects. At the same time, the «train» class (75.13%) shows the highest Recall indicator, which indicates a high level of detection of this object in the test set.
As can be seen from Figure 6, the YOLOv11X neural network model demonstrates aboveaverage prediction accuracy for each class, but for some classes, such as «bicycle» and «traffic light», it demonstrates lower values than for other classes, which is associated with the complexity of their detection. However, if we evaluate the model in the context of 80-class classification, the calculated indicators are good.
For the classification of named persons, the CNN neural network model was trained to identify 19 named persons and a separate class was «other», in which photos of different persons were mixed, which were taken in the amount of 50 photos from the dataset [47]. To train the neural network model, 50 face frames were taken from the video stream for each person in different positions using the software prototype interface. The total number of samples is 1000 images and all classes were used equally in the training process.
The influence of batch size and number of epochs when training the CNN model on the accuracy of classification of named persons is given in Table 2. The following macrometric indicators were obtained: Accuracy 97%, Precision 95%, Recall 96%, F1-score 95%.
Micrometrics Accuracy, Precision, Recall, F1-score are given in Table 3.
According to the results presented in Table 3, the optimal values of batch size are 32 and the number of epochs is 16, since with these parameters the highest classification accuracy of 97% is achieved.
The results of the CNN neural network model for the classification of named persons
demonstrate high accuracy. In particular, the micrometrics Accuracy, Precision, Recall and F1-score
indicate sufficient accuracy and completeness of recognition for each class. The obtained neural
network model has higher accuracy rates, compared to [
The limitations of the used CNN model are that it demonstrates a decrease in the level of confidence in the classification of named persons with a significant head rotation, in particular, if the face is in a 3/4 angle or in profile. In such cases, the model cannot identify a person due to the lack of a sufficient number of recognition features. Also, glasses and masks reduce the classification confidence score, but headgear does not affect the classification accuracy.
Another major limitation is the lack of an algorithm for measuring the distance to objects in the frame. Since the model analyzes two-dimensional images, there is a problem with correctly identifying objects that are either too close or too far from the camera. Also, due to the distortion of the perspective of objects in the frame and the loss of details, such objects may be incorrectly classified or even missed. For example, a car that is at a considerable distance may be recognized as a different class due to its reduced size and distortion of proportions. An example of incorrect classification of a car is shown in Figure 7. This is also especially critical for named persons, since the accuracy of their classification directly depends on the quality and size of the face in the frame. а) b)
Another limitation is the frame refresh rate and the creation of the current audio stream. The delay between frame processing can cause situations where the object has already left the field of view, but its sound is still being played. This can create problems for users, since the audio stream will not be synchronized with the current situation in the environment.
The prospect of further research is to study the method in real conditions to ensure the optimal delay of the audio stream output. As well as the use of sensors to estimate the distance to objects, this will compensate for the scaling problem to avoid false classifications for objects that are too close or far from the device camera.
Method of augmented reality audiostream creation using CNN is proposed, which can increase inclusion and safety for people with visual impairments due to additional classification of named persons and priority audio output of dangerous objects into the audio stream.
To identify objects in the video stream, the YOLOv11x neural network model was used, which allows in real time by analyzing the video stream received by the camera, to identify and classify objects from the environment, both outdoors and indoors. To classify named persons identified by the YOLOv11x model, a cascade classifier for face identification and a CNN neural network model trained for multi-class classification of named persons were used.
For applied research of the method proposed in paper, test software was developed and comparison of neural network models YOLO5n, YOLOv5x, YOLOv8n, YOLOv8x, YOLOv11n, and YOLOv11x was performed. By analyzing the obtained results, namely the macrometrics Precision, Recall, mAP@0.5, mAP@0.5:0.95 and FPS, it was determined that the YOLOv11x model is better suited for identifying the classification of objects in the environment, as it received higher indicators. In particular, the Accuracy value is 74.1%, Precision is 73.74%, which indicates the high accuracy of the model in assigning the detected objects to the correct classes. Recall, which is 65.85%, reflects the ability of the model to find a significant proportion of all objects present, although some of them may remain undetected. The mAP@0.5 value, which is 71.34%, indicates the overall quality of the model at a relatively low overlap threshold, and mAP@0.5:0.95, which is 54.85%, demonstrates its accuracy at different overlap thresholds. The calculated FPS of the YOLOv5x, YOLOv8x, YOLOv11x models also indicates that YOLOv11x, compared to others, is able to process the same number of frames in a shorter period of time.
The obtained results of macrometrics and micrometrics Accuracy, Precision, Recall and F1-score of the CNN neural network model for the task of classification of named persons demonstrate high classification accuracy. Macrometrics (Accuracy 97%, Precision 95%, Recall 96%, F1-score 95%) indicate a general high accuracy of classification of named persons. Minor variations in the indicators for individual classes indicate a minimum number of errors, which does not significantly affect the overall accuracy of classification of named persons.
Proposed method has limitation in that there is no algorithm for measuring the distance to objects, which leads to a decrease in the accuracy of their identification and classification, especially for distant objects. In addition, limited frame refresh rate can cause delay in the audio stream, due to which objects are voiced with delay or after they disappear from the view field.
Therefore, the method presented in the paper can increase the inclusion and safety of people with visual impairments. Developed method differs from existing ones by able to classify named individuals in the video stream for boosting the inclusion of visually impaired people; also, method prioritizes dangerous objects when generating audiostream for boosting the safety of visually impaired people. Considering the above, the method of augmented reality audiostream creation using CNN will contribute to achieving UNDP SDG No. 3, No. 10 and No. 11.
The authors have not employed any Generative AI tools. [43] O. Sobko, O. Mazurets, M. Molchanova, I. Krak, O. Barmak, Method for analysis and formation of representative text datasets, CEUR Workshop Proceedings, 3899 (2024) 84-98. URL: https://ceur-ws.org/Vol-3899/paper9.pdf. [44] Y. Krak, O. Barmak, O. Mazurets, The Practice Investigation of the Information Technology Efficiency for Automated Definition of Terms in the Semantic Content of Educational Materials, CEUR Workshop Proceedings, 163 (2016) 237–245. URL: doi:10.15407/pp2016.0203.237. [45] O. Zalutska, M. Molchanova, O. Sobko, O. Mazurets, O. Pasichnyk, O. Barmak, I. Krak, Method for Sentiment Analysis of Ukrainian-Language Reviews in E-Commerce Using RoBERTa Neural Network, CEUR Workshop Proceedings, 3387 (2023) 344-356. URL: https://ceurws.org/Vol-3387/paper26.pdf. [46] TechChannel00001, YouTube Video, 2025. URL: https://www.youtube.com/watch?v=Gr0HpDM8Ki8. [47] Kaggle, LFW People Dataset, 2025. URL: https://www.kaggle.com/datasets/atulanandjha/lfwpeople.