The research focuses on addressing the challenges faced by visually impaired individuals in identifying banknotes in the city of Arequipa. The development of an assistance device based on computer vision and IoT is proposed to help these individuals recognize different denominations of banknotes, as well as nearby objects. The state of the art in banknote and object recognition systems is reviewed globally and nationally, highlighting advances in technologies such as machine learning and computer vision. The study follows a Design Thinking approach, including empathy, definition, ideation, prototyping, and evaluation phases. The actions for creating a dataset of banknote images and implementing the realtime vision module in the device are detailed. Although tests with end-users are pending, data has been collected to identify areas for improvement in banknote and nearby object recognition. The research aims to improve the quality of life for visually impaired people in Arequipa by facilitating the identification of banknotes and objects through an economical assistance device based on computer vision and IoT.
Visual impairment is a condition that presents significant challenges in the daily lives of those
who experience it. In the city of Arequipa [
In the current context, technology plays a fundamental role in improving the quality of life for people with visual disabilities. One of the key challenges faced by these people in their daily lives is the identification and management of different denominations of banknotes, a crucial task in financial transactions and daily activities. In response to this need, various global and national research efforts have been carried out to develop banknote recognition systems using advanced technologies such as machine learning, computer vision, and image processing.
Globally, several research studies have focused on the development of banknote recognition systems to assist visually impaired individuals in identifying different denominations of currency.
In [
On the other hand, in [
Conversely, [
In [
In the research [
In [
Finally, in [9], they introduced an AI-backed mobile application for the recognition of banknotes from the United Arab Emirates, aimed at visually impaired people. The application used a pre-trained convolutional neural network to detect and classify banknotes, in addition to providing auditory signals. Although the average accuracy reached 70% in tests and 88% in fivefold cross-validation, the application represents a step toward independence in daily financial transactions.
On a national level, in Peru, research has focused on the design and development of low-cost ergonomic tools to improve the mobility of people with visual disabilities.
In Lima, [10] proposed the creation of an ergonomic GPS-enabled cane for blind people with the aim of increasing their autonomy. The methodology was based on systems engineering for monitoring and software development. The results indicated that the ergonomic cane improved the mobility of people with visual disabilities and allowed tracking by their family members. The importance of considering aspects such as the shape, size, and weight of the cane to achieve the desired ergonomics was highlighted.
On the other hand, in Chiclayo, [11] focused on the development of an Intelligent Geolocating Sensor Cane to support blind people in their mobility. The methodology used was based on the Rational Unified Process (RUP) and embedded systems. The results demonstrated that this sensor cane could make life more dynamic and secure for people with visual disabilities. The fusion of the RUP and embedded systems methodologies contributed to the efficient design of the prototype.
Finally, in Arequipa, [12], the focus was on improving the quality of daily mobility for people with visual disabilities through an Electronic Cane with ultrasonic sensors. The methodology included a descriptive-explanatory study and the experimental design of sensors. The results highlighted that this Electronic Cane could reduce accidents in the daily mobility of people with visual disabilities.
In summary, notable advances have been made worldwide and nationally in creating tools and systems to improve the lives of people with visual disabilities, using technologies such as machine learning, computer vision, and ultrasonic sensors. These advances demonstrate the potential of technology to increase the independence and safety of people with visual disabilities. The research conducted in Peru emphasizes the importance of considering the specific needs of this group and the utility of combining various methodologies to create effective solutions. Together, these developments offer a promising outlook in which technology will continue to play a fundamental role in improving the quality of life for people with visual disabilities, with possibilities for application elsewhere and a focus on inclusion and autonomy.
The study employed an iterative, action-oriented methodology and followed the five-step process of the Design Thinking approach (i.e., empathize, define, ideate, prototype, and evaluate) [16].
This phase was rigorously executed, encompassing a comprehensive literature review of available assistive tools and technologies. Requirements gathering was done through interviews and analysis of local statistics, the evaluation of Artificial Vision algorithms, a detailed comparison of IoT devices, and the meticulous selection of Computer Vision techniques. This ensured that the device would be designed with a closer understanding of the needs and conditions of visually impaired individuals from the Association of the Blind in Arequipa. Best practices and accessible technologies were leveraged to provide an effective solution tailored to their primary needs, which revolve around the recognition of banknotes and nearby everyday objects.
After gaining an empathetic understanding of the main needs of visually impaired individuals, the define phase involved categorizing the results obtained from the evaluation of techniques, Artificial Vision algorithms, and IoT devices. This classification identified and selected the most appropriate and effective approaches to address the needs of visually impaired individuals in the study region. Additionally, a comprehensive compilation of results related to the current situation of these individuals and available assistive tools in the context of Arequipa was conducted. These findings provided a solid foundation for decision-making in the design of the device. It is worth mentioning that all these results were thoroughly documented in the thesis report, thus establishing a solid and well-founded basis for the subsequent ideation and prototyping phases.
With a defined understanding of the problem, a creative flow of ideas is generated. In this phase, a plan is designed and developed to achieve the goal of creating a product that improves or solves the identified problem. To accomplish this, a set of actions was followed for creating the dataset of banknote images. This included setting up the necessary image capture equipment, involving the selection of appropriate cameras and sensors. A meticulous procedure for image capture was developed, considering a variety of relevant scenarios and situations reflecting the diversity of environments in which visually impaired individuals might use the device. Image capture was carried out extensively, covering banknotes representative of the local reality. Subsequently, labeling of these images was performed, preparing them for use in training the Artificial Vision module. The results obtained in this stage provide a solid dataset of banknote images, crucial for the development of the product. These achievements have been thoroughly documented, establishing a clear and well-founded direction for the subsequent prototyping and evaluation stages.
During this stage, the implementation of the vision module designed for real-time image capture and processing, a fundamental element of the proposal, has taken place. This achievement was not only crucial for the device's operation but also highlights the robustness of the strategy in combining Internet of Things (IoT) technologies with advanced Computer Vision techniques to accomplish the task of recognizing banknotes and everyday objects near the user. Additionally, exhaustive tests have been conducted, and the components of the vision module have been debugged. This has allowed for the effective identification and addressing of any obstacles or potential deficiencies, ensuring optimal system performance. Furthermore, acceptable communication has been achieved between the vision module and the IoT device, thereby reinforcing the reliability and effectiveness of the proposed solution. The design of the first prototype is presented below:
In the evaluation phase, technical and functional tests have been conducted under controlled conditions to validate the integration of the system, measure effectiveness in banknote recognition, and collect technical feedback. Although end-users have not yet been included in these evaluations, crucial data has been gathered to identify areas for improvement in the design and operation of the device. These data and observations will serve as a foundation for future tests with end-users and contribute to the overall success of the project.
One of the main outcomes in the development of the project proposal is: 1. The elaboration of a systematic review of assistive tools for visually impaired individuals, conducted during the empathize stage, allowed for the identification of the main limitations and contributions in the last 5 years of research in the field. These are related to advances in assistive technologies, the Internet of Things, and Computer Vision. In figures 4 and 5, the main findings of the systematic review are presented, emphasizing the key limitations and contributions. 2. The creation of the dataset for Peruvian banknotes, along with its publication on the IEEE DataPort platform. It resulted in a total of 9315 processed images, with 6568 for training, 2486 for validation, and 261 for testing. It covers a total of 16 different categories, including both the obverse and reverse sides of 10, 20, 50, and 100 Peruvian soles from both old and new bill families, spanning from 2011 to 2021. The images were captured using rear cameras of mobile phones under various backgrounds and lighting conditions, including cluttered backgrounds and images of folded banknotes.
Access link: https://ieee-dataport.org/documents/dataset-peruvian-banknotes 3. The construction of the first prototype of the proposed assistive device is a significant outcome of the project. With this, the overall objective of providing assistance in the recognition of Peruvian banknotes and nearby objects to the user would be achieved. As a key contribution within the construction of the first prototype, the effective use of pre-trained deep learning models for banknotes and models pre-trained on an ImageNet dataset for objects, the characterization of processing complexity, the design of an economical assistive system, and the empirical evaluation of accuracy in real-world conditions have been highlighted. Thus, achieving the design of the first prototype, the optimal processing of Peruvian banknote and object images, and the empirical evaluation of the accuracy of pre-trained deep learning algorithms with 68% for banknote recognition and 73% for object recognition with Fine-Tuning.
This study has focused on addressing the challenges faced by visually impaired individuals in identifying banknotes in the city of Arequipa and the development of an assistive device based on artificial vision and IoT to mitigate these challenges. From the research and work conducted, the following conclusions can be drawn: • The need for technological solutions for visually impaired individuals is critical. The identification of banknotes and objects is essential in daily life, making the creation of economical and effective assistive devices imperative to improve their quality of life. • The exhaustive review of the state of the art in global and national banknote recognition systems highlights that technology, particularly machine learning, computer vision, and sensors, can be a valuable ally for visually impaired individuals, providing them with independence and security. • The Design Thinking approach, encompassing the stages of empathy, definition, ideation, prototyping, and evaluation, has proven effective for the development of an assistive device that caters to the specific needs and conditions of visually impaired individuals in Arequipa. • The creation of a dataset of banknote images and the development of the real-time vision module are fundamental achievements supporting the viability and effectiveness of the proposed assistive device. • Although tests with end-users are still pending, the technical data collected so far will serve as a basis for future evaluations and refinements of the device. This project is anticipated not only to enhance the quality of life for visually impaired individuals in Arequipa but also to potentially serve as a model for similar solutions worldwide. [9] A. Khalil, M. Yaghi, T. Basmaji, M. Faizal, Z. Farhan, A. Ali y M. Ghazal, «Mobile Deep Classification of UAE Banknotes for the Visually Challenged» 2022 9th International Conference on Future Internet of Things and Cloud (FiCloud), pp. 321-325, 2022. [10] E. Vela, «Diseño e implementación de un bastón ergonómico con sistema de posicionamiento global para mejorar el desplazamiento de personas invidentes en el centro “la unión nacional de ciegos del Perú",» Ingeniería Electrónica, Lima, 2019. [11] M. Vilchez, «Bastón sensorial geolocalizador inteligente para apoyar en el desplazamiento de personas invidentes en la organización regional de ciegos del Perú – Chiclayo,» Ingeniería de Sistemas y Computación, Chiclayo, 2021. [12] C. Lizárraga, «Propuesta para el diseño de un bastón electrónico para personas invidentes que mejorara la calidad de su desplazamiento diario,» Ingeniería Industrial, Arequipa, 2018. [13] H. Shah, M. Amin, K. Dadwani, N. Desai, A. Chatiwala, T. Senjyu, C. So-In y A. Joshi, «Real-Time Object Detection System with Voice Feedback for the Blind People» Smart Trends in Computing and Communications, pp. 683-690, 2023. [14] Z. Yuan, T. Azzino, Y. a. L. Y. Hao, H. Pei, A. Boldini, M. Mezzavilla, M. Beheshti, M. Porfiri, T. E.
Hudson, W. Seiple, Y. Fang, S. Rangan, Y. Wang y J.-R. Rizzo, «Network-Aware 5G Edge Computing for Object Detection: Augmenting Wearables to “See” More, Farther and Faster» IEEE Access, vol. 10, pp. 29612-29632, 2022. [15] M. Valipoor y A. de Antonio, «Recent trends in computer vision-driven scene understanding for VI/blind users: a systematic mapping» Universal Access in the Information Society, 2022. [16] P. Geerts, J. Eijsink, A. Moser, P. ter Horst, C. Boersma y M. Postma, «Rationale and development of an e-health application to deliver patient-centered care during treatment for recently diagnosed multiple myeloma patients: pilot study of the MM E-coach» Pilot and Feasibility Studies, vol. 9, nº 1, 2023. [17] N. Caytuiro-Silva, J. Peña-Alejandro y E. Castro-Gutierrez, «DATASET OF PERUVIAN BANKNOTES,» IEEE DataPort, Arequipa, 2023.