Proposal Based on Computer Vision and IoT for the Development of an Ergonomic and Low-Cost Assistance Device for People with Visual Disabilities Nicolás E. Caytuiro-Silva1, Eveling G. Castro-Gutierrez1 and Jackeline M. Peña-Alejandro1 1 Universidad Católica de Santa María, Urb. San José s/n Umacollo, Arequipa, Perú Abstract The research focuses on addressing the challenges faced by visually impaired individuals in identifying banknotes in the city of Arequipa. The development of an assistance device based on computer vision and IoT is proposed to help these individuals recognize different denominations of banknotes, as well as nearby objects. The state of the art in banknote and object recognition systems is reviewed globally and nationally, highlighting advances in technologies such as machine learning and computer vision. The study follows a Design Thinking approach, including empathy, definition, ideation, prototyping, and evaluation phases. The actions for creating a dataset of banknote images and implementing the real- time vision module in the device are detailed. Although tests with end-users are pending, data has been collected to identify areas for improvement in banknote and nearby object recognition. The research aims to improve the quality of life for visually impaired people in Arequipa by facilitating the identification of banknotes and objects through an economical assistance device based on computer vision and IoT. Keywords Computer Vision, IoT, Assistance Device, Economical, Low-cost, Visual Impairment. 1 1. Introduction Visual impairment is a condition that presents significant challenges in the daily lives of those who experience it. In the city of Arequipa [1], as in many other regions, people with visual disabilities face additional difficulties when trying to identify banknotes and objects in their everyday environment. This project aims to address this issue by developing an economical assistance device based on computer vision and IoT. This device will enable visually impaired individuals to identify banknotes and objects, providing them with greater autonomy and independence in their daily lives. In this context, the design, implementation, and evaluation of this innovative device are presented, which has the potential to improve the quality of life for visually impaired people in Arequipa and serve as a model for similar solutions worldwide [2]. 2. State of the art In the current context, technology plays a fundamental role in improving the quality of life for people with visual disabilities. One of the key challenges faced by these people in their daily lives is the identification and management of different denominations of banknotes, a crucial task in financial transactions and daily activities. In response to this need, various global and national research efforts have been carried out to develop banknote recognition systems using advanced technologies such as machine learning, computer vision, and image processing. Globally, several research studies have focused on the development of banknote recognition systems to assist visually impaired individuals in identifying different denominations of currency. JINIS 2023: XXX International Conference on Systems Engineering, October 03–05, 2023, Arequipa, Peru nicolas.caytuiro@ucsm.edu.pe (N. Caytuiro); ecastrog@ucsm.edu.pe (E. Castro); jackeline.pena@ucsm.edu.pe (J. Peña) 0000-0003-1656-396X (N. Caytuiro); 0000-0002-0203-041X (E. Castro); 0000-0002-3586-1826 (J. Peña) © 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings (CEUR-WS.org) CEUR ht tp: // ceur -ws .or g Works hop I SSN1613- 0073 Pr oceedi ngs CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings In [3], they proposed developing a Malaysian banknote recognition system to assist visually impaired people. The study aimed to analyze the impact of region and orientation on the performance of Machine Learning and Deep Learning approaches. The results revealed that SVM and BC algorithms achieved 100% accuracy, while kNN and DTC achieved 99.7%. It was also concluded that orientation influences the performance of the AlexNet model, showing better execution with similar orientation data. On the other hand, in [4], the accurate classification of Honduran banknotes was the focus, including a new L200 banknote, with an emphasis on adapting to the incorporation of new banknotes in circulation. Two high-performance methods were presented. The first relied on advanced local descriptors such as ORB or SIFT, generating feature vectors for algorithms like SVM and Random Forests. The second introduced the LempiraNet CNN, which used transfer learning to address data limitations. The results demonstrated outstanding accuracy of 98% or higher, with LempiraNet being significantly faster than the other method. Conversely, [5] proposed an innovative approach based on quaternionic wavelet transform (QWT) and a deep convolutional neural network for banknote classification. This methodology leveraged the multiscale structure and directional sensitivity of QWT. The results highlighted superior performance compared to other state-of-the-art banknote classification algorithms, as well as meeting real-time requirements for banknote classification systems. In [6], the focus was on creating an Iraqi banknote classification system based on Deep Learning and computer vision technology. The central objective was to develop a multiclass classification model capable of distinguishing between different denominations of Iraqi banknotes and providing equivalent voice commands to inform visually impaired people about the value of the banknotes. The system achieved an impressive accuracy of 98.6%, demonstrating its viability and potential to enhance the financial independence of this user group. In the research [7], they focused on the recognition of Ethiopian banknotes using a convolutional neural network (CNN). The research included comprehensive evaluations of different CNN architectures and optimization techniques. The highlighted architecture, MobileNetV2, implemented with RMSProp optimization, achieved outstanding accuracy of 96.4%. Additionally, the model was implemented on an integrated platform using Raspberry Pi, with potential applications in automatic monetary transactions. In [8], they addressed the recognition of Colombian banknotes by visually impaired people. They developed a classification system for eleven denominations of banknotes using image processing techniques and MLP neural networks. The system achieved 95% accuracy after manipulating samples and expanding the dataset, underscoring its effectiveness in the autonomy of visually impaired individuals. Finally, in [9], they introduced an AI-backed mobile application for the recognition of banknotes from the United Arab Emirates, aimed at visually impaired people. The application used a pre-trained convolutional neural network to detect and classify banknotes, in addition to providing auditory signals. Although the average accuracy reached 70% in tests and 88% in five- fold cross-validation, the application represents a step toward independence in daily financial transactions. On a national level, in Peru, research has focused on the design and development of low-cost ergonomic tools to improve the mobility of people with visual disabilities. In Lima, [10] proposed the creation of an ergonomic GPS-enabled cane for blind people with the aim of increasing their autonomy. The methodology was based on systems engineering for monitoring and software development. The results indicated that the ergonomic cane improved the mobility of people with visual disabilities and allowed tracking by their family members. The importance of considering aspects such as the shape, size, and weight of the cane to achieve the desired ergonomics was highlighted. On the other hand, in Chiclayo, [11] focused on the development of an Intelligent Geolocating Sensor Cane to support blind people in their mobility. The methodology used was based on the Rational Unified Process (RUP) and embedded systems. The results demonstrated that this sensor cane could make life more dynamic and secure for people with visual disabilities. The fusion of the RUP and embedded systems methodologies contributed to the efficient design of the prototype. Finally, in Arequipa, [12], the focus was on improving the quality of daily mobility for people with visual disabilities through an Electronic Cane with ultrasonic sensors. The methodology included a descriptive-explanatory study and the experimental design of sensors. The results highlighted that this Electronic Cane could reduce accidents in the daily mobility of people with visual disabilities. In summary, notable advances have been made worldwide and nationally in creating tools and systems to improve the lives of people with visual disabilities, using technologies such as machine learning, computer vision, and ultrasonic sensors. These advances demonstrate the potential of technology to increase the independence and safety of people with visual disabilities. The research conducted in Peru emphasizes the importance of considering the specific needs of this group and the utility of combining various methodologies to create effective solutions. Together, these developments offer a promising outlook in which technology will continue to play a fundamental role in improving the quality of life for people with visual disabilities, with possibilities for application elsewhere and a focus on inclusion and autonomy. 3. Materials and Methods The study employed an iterative, action-oriented methodology and followed the five-step process of the Design Thinking approach (i.e., empathize, define, ideate, prototype, and evaluate) [16]. Figure 1: Design Thinking Methodology [16] 3.1. Phase 01: Empathize This phase was rigorously executed, encompassing a comprehensive literature review of available assistive tools and technologies. Requirements gathering was done through interviews and analysis of local statistics, the evaluation of Artificial Vision algorithms, a detailed comparison of IoT devices, and the meticulous selection of Computer Vision techniques. This ensured that the device would be designed with a closer understanding of the needs and conditions of visually impaired individuals from the Association of the Blind in Arequipa. Best practices and accessible technologies were leveraged to provide an effective solution tailored to their primary needs, which revolve around the recognition of banknotes and nearby everyday objects. 3.2. Phase 02: Define After gaining an empathetic understanding of the main needs of visually impaired individuals, the define phase involved categorizing the results obtained from the evaluation of techniques, Artificial Vision algorithms, and IoT devices. This classification identified and selected the most appropriate and effective approaches to address the needs of visually impaired individuals in the study region. Additionally, a comprehensive compilation of results related to the current situation of these individuals and available assistive tools in the context of Arequipa was conducted. These findings provided a solid foundation for decision-making in the design of the device. It is worth mentioning that all these results were thoroughly documented in the thesis report, thus establishing a solid and well-founded basis for the subsequent ideation and prototyping phases. 3.3. Phase 03: Ideate With a defined understanding of the problem, a creative flow of ideas is generated. In this phase, a plan is designed and developed to achieve the goal of creating a product that improves or solves the identified problem. To accomplish this, a set of actions was followed for creating the dataset of banknote images. This included setting up the necessary image capture equipment, involving the selection of appropriate cameras and sensors. A meticulous procedure for image capture was developed, considering a variety of relevant scenarios and situations reflecting the diversity of environments in which visually impaired individuals might use the device. Image capture was carried out extensively, covering banknotes representative of the local reality. Subsequently, labeling of these images was performed, preparing them for use in training the Artificial Vision module. The results obtained in this stage provide a solid dataset of banknote images, crucial for the development of the product. These achievements have been thoroughly documented, establishing a clear and well-founded direction for the subsequent prototyping and evaluation stages. Table 1 Total Number of Captured Banknote Images Denomination Total 10 20 50 100 New Banknote Family 728 894 1190 939 3751 Old Banknote Family 1025 1162 1831 1285 5303 Total 1753 2056 3021 2224 6054 3.4. Phase 04: Prototyping During this stage, the implementation of the vision module designed for real-time image capture and processing, a fundamental element of the proposal, has taken place. This achievement was not only crucial for the device's operation but also highlights the robustness of the strategy in combining Internet of Things (IoT) technologies with advanced Computer Vision techniques to accomplish the task of recognizing banknotes and everyday objects near the user. Additionally, exhaustive tests have been conducted, and the components of the vision module have been debugged. This has allowed for the effective identification and addressing of any obstacles or potential deficiencies, ensuring optimal system performance. Furthermore, acceptable communication has been achieved between the vision module and the IoT device, thereby reinforcing the reliability and effectiveness of the proposed solution. The design of the first prototype is presented below: Figure 2: Front Design of the Prototype Figure 3: Back Design of the Prototype 3.5. Phase 05: Evaluation In the evaluation phase, technical and functional tests have been conducted under controlled conditions to validate the integration of the system, measure effectiveness in banknote recognition, and collect technical feedback. Although end-users have not yet been included in these evaluations, crucial data has been gathered to identify areas for improvement in the design and operation of the device. These data and observations will serve as a foundation for future tests with end-users and contribute to the overall success of the project. 4. Results One of the main outcomes in the development of the project proposal is: 1. The elaboration of a systematic review of assistive tools for visually impaired individuals, conducted during the empathize stage, allowed for the identification of the main limitations and contributions in the last 5 years of research in the field. These are related to advances in assistive technologies, the Internet of Things, and Computer Vision. In figures 4 and 5, the main findings of the systematic review are presented, emphasizing the key limitations and contributions. Figure 4: Limitations from the Systematic Review Figure 5: Contributions from the Systematic Review 2. The creation of the dataset for Peruvian banknotes, along with its publication on the IEEE DataPort platform. It resulted in a total of 9315 processed images, with 6568 for training, 2486 for validation, and 261 for testing. It covers a total of 16 different categories, including both the obverse and reverse sides of 10, 20, 50, and 100 Peruvian soles from both old and new bill families, spanning from 2011 to 2021. The images were captured using rear cameras of mobile phones under various backgrounds and lighting conditions, including cluttered backgrounds and images of folded banknotes. Access link: https://ieee-dataport.org/documents/dataset-peruvian-banknotes 3. The construction of the first prototype of the proposed assistive device is a significant outcome of the project. With this, the overall objective of providing assistance in the recognition of Peruvian banknotes and nearby objects to the user would be achieved. As a key contribution within the construction of the first prototype, the effective use of pre-trained deep learning models for banknotes and models pre-trained on an ImageNet dataset for objects, the characterization of processing complexity, the design of an economical assistive system, and the empirical evaluation of accuracy in real-world conditions have been highlighted. Thus, achieving the design of the first prototype, the optimal processing of Peruvian banknote and object images, and the empirical evaluation of the accuracy of pre-trained deep learning algorithms with 68% for banknote recognition and 73% for object recognition with Fine-Tuning. 5. Conclusions This study has focused on addressing the challenges faced by visually impaired individuals in identifying banknotes in the city of Arequipa and the development of an assistive device based on artificial vision and IoT to mitigate these challenges. From the research and work conducted, the following conclusions can be drawn: • The need for technological solutions for visually impaired individuals is critical. The identification of banknotes and objects is essential in daily life, making the creation of economical and effective assistive devices imperative to improve their quality of life. • The exhaustive review of the state of the art in global and national banknote recognition systems highlights that technology, particularly machine learning, computer vision, and sensors, can be a valuable ally for visually impaired individuals, providing them with independence and security. • The Design Thinking approach, encompassing the stages of empathy, definition, ideation, prototyping, and evaluation, has proven effective for the development of an assistive device that caters to the specific needs and conditions of visually impaired individuals in Arequipa. • The creation of a dataset of banknote images and the development of the real-time vision module are fundamental achievements supporting the viability and effectiveness of the proposed assistive device. • Although tests with end-users are still pending, the technical data collected so far will serve as a basis for future evaluations and refinements of the device. This project is anticipated not only to enhance the quality of life for visually impaired individuals in Arequipa but also to potentially serve as a model for similar solutions worldwide. References [1] C. Lizarraga, «Propuesta para el diseño de un bastón electrónico para personas invidentes que mejorara la calidad de su desplazamiento diario,» Universidad Continental, Arequipa, 2018. [2] C. Killa, «Pulsera para guiar a personas con discapacidad visual recibe medalla de oro en Corea,» Agencia Peruana de Noticias, 17 Octubre 2021. [En línea]. Available: https://andina.pe/agencia/noticia-pulsera-para-guiar-a-personas-discapacidad-visual- recibe-medalla-oro-corea-865204.aspx. [Último acceso: 15 Septiembre 2023]. [3] N. Sufri, N. Rahmad, N. Ghazali, N. Shahar y M. Asári, «Vision Based System for Banknote Recognition Using Different Machine Learning and Deep Learning Approach» 2019 IEEE 10th Control and System Graduate Research Colloquium (ICSGRC), pp. 5-8, 2019. [4] S. Castelar, L. A. Banegas, D. A. Mendoza, J. C. Soto y K. Davila, «Automated Honduran Banknote Image Classification using Machine Learning» 2022 IEEE 40th Central America and Panama Convention (CONCAPAN), pp. 1-6, 2022. [5] X. Huang y S. Gai, «Banknote Classification Based on Convolutional Neural Network in Quaternion Wavelet Domain» IEEE Access, vol. 8, pp. 162141-162148, 2020. [6] S. R. Awad, B. T. Sharef, A. M. Salih y F. L. Malallah, «Deep Learning-Based Iraqi Banknotes Classification System for Blind People» Eastern-European Journal of Enterprise Technologies, vol. 1, nº 115, pp. 31-38, 2022. [7] C.-B. Fan, D. T. Aseffa, H. Kalla y S. Mishra, «Ethiopian Banknote Recognition Using Convolutional Neural Network and Its Prototype Development Using Embedded Platform» Journal of Sensors, vol. 2022, p. 4505089, 2022. [8] K. Tamayo, «Sistema de Reconocimiento de Billetes para Personas con Discapacidad Visual Mediante Visión Artificial,» Universidad EIA, Colombia, 2022. [9] A. Khalil, M. Yaghi, T. Basmaji, M. Faizal, Z. Farhan, A. Ali y M. Ghazal, «Mobile Deep Classification of UAE Banknotes for the Visually Challenged» 2022 9th International Conference on Future Internet of Things and Cloud (FiCloud), pp. 321-325, 2022. [10] E. Vela, «Diseño e implementación de un bastón ergonómico con sistema de posicionamiento global para mejorar el desplazamiento de personas invidentes en el centro “la unión nacional de ciegos del Perú",» Ingeniería Electrónica, Lima, 2019. [11] M. Vilchez, «Bastón sensorial geolocalizador inteligente para apoyar en el desplazamiento de personas invidentes en la organización regional de ciegos del Perú – Chiclayo,» Ingeniería de Sistemas y Computación, Chiclayo, 2021. [12] C. Lizárraga, «Propuesta para el diseño de un bastón electrónico para personas invidentes que mejorara la calidad de su desplazamiento diario,» Ingeniería Industrial, Arequipa, 2018. [13] H. Shah, M. Amin, K. Dadwani, N. Desai, A. Chatiwala, T. Senjyu, C. So-In y A. Joshi, «Real-Time Object Detection System with Voice Feedback for the Blind People» Smart Trends in Computing and Communications, pp. 683-690, 2023. [14] Z. Yuan, T. Azzino, Y. a. L. Y. Hao, H. Pei, A. Boldini, M. Mezzavilla, M. Beheshti, M. Porfiri, T. E. Hudson, W. Seiple, Y. Fang, S. Rangan, Y. Wang y J.-R. Rizzo, «Network-Aware 5G Edge Computing for Object Detection: Augmenting Wearables to “See” More, Farther and Faster» IEEE Access, vol. 10, pp. 29612-29632, 2022. [15] M. Valipoor y A. de Antonio, «Recent trends in computer vision-driven scene understanding for VI/blind users: a systematic mapping» Universal Access in the Information Society, 2022. [16] P. Geerts, J. Eijsink, A. Moser, P. ter Horst, C. Boersma y M. Postma, «Rationale and development of an e-health application to deliver patient-centered care during treatment for recently diagnosed multiple myeloma patients: pilot study of the MM E-coach» Pilot and Feasibility Studies, vol. 9, nº 1, 2023. [17] N. Caytuiro-Silva, J. Peña-Alejandro y E. Castro-Gutierrez, «DATASET OF PERUVIAN BANKNOTES,» IEEE DataPort, Arequipa, 2023.