=Paper=
{{Paper
|id=Vol-3693/paper15
|storemode=property
|title=Proposal Based on Computer Vision and IoT for the Development of an Ergonomic and Low-Cost Assistance Device for People with Visual Disabilities
|pdfUrl=https://ceur-ws.org/Vol-3693/paper15.pdf
|volume=Vol-3693
|authors=Nicolás E. Caytuiro-Silva,Eveling G. Castro-Gutierrez,Jackeline M. Peña-Alejandro
|dblpUrl=https://dblp.org/rec/conf/jinis/Caytuiro-SilvaG23a
}}
==Proposal Based on Computer Vision and IoT for the Development of an Ergonomic and Low-Cost Assistance Device for People with Visual Disabilities==
https://ceur-ws.org/Vol-3693/paper15.pdf
Proposal Based on Computer Vision and IoT for the
Development of an Ergonomic and Low-Cost Assistance
Device for People with Visual Disabilities
Nicolás E. Caytuiro-Silva1, Eveling G. Castro-Gutierrez1 and Jackeline M. Peña-Alejandro1
1 Universidad Católica de Santa María, Urb. San José s/n Umacollo, Arequipa, Perú
Abstract
The research focuses on addressing the challenges faced by visually impaired individuals in identifying
banknotes in the city of Arequipa. The development of an assistance device based on computer vision
and IoT is proposed to help these individuals recognize different denominations of banknotes, as well
as nearby objects. The state of the art in banknote and object recognition systems is reviewed globally
and nationally, highlighting advances in technologies such as machine learning and computer vision.
The study follows a Design Thinking approach, including empathy, definition, ideation, prototyping, and
evaluation phases. The actions for creating a dataset of banknote images and implementing the real-
time vision module in the device are detailed. Although tests with end-users are pending, data has been
collected to identify areas for improvement in banknote and nearby object recognition. The research
aims to improve the quality of life for visually impaired people in Arequipa by facilitating the
identification of banknotes and objects through an economical assistance device based on computer
vision and IoT.
Keywords
Computer Vision, IoT, Assistance Device, Economical, Low-cost, Visual Impairment. 1
1. Introduction
Visual impairment is a condition that presents significant challenges in the daily lives of those
who experience it. In the city of Arequipa [1], as in many other regions, people with visual
disabilities face additional difficulties when trying to identify banknotes and objects in their
everyday environment. This project aims to address this issue by developing an economical
assistance device based on computer vision and IoT. This device will enable visually impaired
individuals to identify banknotes and objects, providing them with greater autonomy and
independence in their daily lives. In this context, the design, implementation, and evaluation of
this innovative device are presented, which has the potential to improve the quality of life for
visually impaired people in Arequipa and serve as a model for similar solutions worldwide [2].
2. State of the art
In the current context, technology plays a fundamental role in improving the quality of life for
people with visual disabilities. One of the key challenges faced by these people in their daily lives
is the identification and management of different denominations of banknotes, a crucial task in
financial transactions and daily activities. In response to this need, various global and national
research efforts have been carried out to develop banknote recognition systems using advanced
technologies such as machine learning, computer vision, and image processing.
Globally, several research studies have focused on the development of banknote recognition
systems to assist visually impaired individuals in identifying different denominations of currency.
JINIS 2023: XXX International Conference on Systems Engineering, October 03–05, 2023, Arequipa, Peru
nicolas.caytuiro@ucsm.edu.pe (N. Caytuiro); ecastrog@ucsm.edu.pe (E. Castro); jackeline.pena@ucsm.edu.pe (J.
Peña)
0000-0003-1656-396X (N. Caytuiro); 0000-0002-0203-041X (E. Castro); 0000-0002-3586-1826 (J. Peña)
© 2023 Copyright for this paper by its authors.
Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR Workshop Proceedings (CEUR-WS.org)
CEUR
ht
tp:
//
ceur
-ws
.or
g
Works
hop I
SSN1613-
0073
Pr
oceedi
ngs
CEUR
ceur-ws.org
Workshop ISSN 1613-0073
Proceedings
In [3], they proposed developing a Malaysian banknote recognition system to assist visually
impaired people. The study aimed to analyze the impact of region and orientation on the
performance of Machine Learning and Deep Learning approaches. The results revealed that SVM
and BC algorithms achieved 100% accuracy, while kNN and DTC achieved 99.7%. It was also
concluded that orientation influences the performance of the AlexNet model, showing better
execution with similar orientation data.
On the other hand, in [4], the accurate classification of Honduran banknotes was the focus,
including a new L200 banknote, with an emphasis on adapting to the incorporation of new
banknotes in circulation. Two high-performance methods were presented. The first relied on
advanced local descriptors such as ORB or SIFT, generating feature vectors for algorithms like
SVM and Random Forests. The second introduced the LempiraNet CNN, which used transfer
learning to address data limitations. The results demonstrated outstanding accuracy of 98% or
higher, with LempiraNet being significantly faster than the other method.
Conversely, [5] proposed an innovative approach based on quaternionic wavelet transform
(QWT) and a deep convolutional neural network for banknote classification. This methodology
leveraged the multiscale structure and directional sensitivity of QWT. The results highlighted
superior performance compared to other state-of-the-art banknote classification algorithms, as
well as meeting real-time requirements for banknote classification systems.
In [6], the focus was on creating an Iraqi banknote classification system based on Deep
Learning and computer vision technology. The central objective was to develop a multiclass
classification model capable of distinguishing between different denominations of Iraqi
banknotes and providing equivalent voice commands to inform visually impaired people about
the value of the banknotes. The system achieved an impressive accuracy of 98.6%, demonstrating
its viability and potential to enhance the financial independence of this user group.
In the research [7], they focused on the recognition of Ethiopian banknotes using a
convolutional neural network (CNN). The research included comprehensive evaluations of
different CNN architectures and optimization techniques. The highlighted architecture,
MobileNetV2, implemented with RMSProp optimization, achieved outstanding accuracy of
96.4%. Additionally, the model was implemented on an integrated platform using Raspberry Pi,
with potential applications in automatic monetary transactions.
In [8], they addressed the recognition of Colombian banknotes by visually impaired people.
They developed a classification system for eleven denominations of banknotes using image
processing techniques and MLP neural networks. The system achieved 95% accuracy after
manipulating samples and expanding the dataset, underscoring its effectiveness in the autonomy
of visually impaired individuals.
Finally, in [9], they introduced an AI-backed mobile application for the recognition of
banknotes from the United Arab Emirates, aimed at visually impaired people. The application
used a pre-trained convolutional neural network to detect and classify banknotes, in addition to
providing auditory signals. Although the average accuracy reached 70% in tests and 88% in five-
fold cross-validation, the application represents a step toward independence in daily financial
transactions.
On a national level, in Peru, research has focused on the design and development of low-cost
ergonomic tools to improve the mobility of people with visual disabilities.
In Lima, [10] proposed the creation of an ergonomic GPS-enabled cane for blind people with
the aim of increasing their autonomy. The methodology was based on systems engineering for
monitoring and software development. The results indicated that the ergonomic cane improved
the mobility of people with visual disabilities and allowed tracking by their family members. The
importance of considering aspects such as the shape, size, and weight of the cane to achieve the
desired ergonomics was highlighted.
On the other hand, in Chiclayo, [11] focused on the development of an Intelligent Geolocating
Sensor Cane to support blind people in their mobility. The methodology used was based on the
Rational Unified Process (RUP) and embedded systems. The results demonstrated that this
sensor cane could make life more dynamic and secure for people with visual disabilities. The
fusion of the RUP and embedded systems methodologies contributed to the efficient design of the
prototype.
Finally, in Arequipa, [12], the focus was on improving the quality of daily mobility for people
with visual disabilities through an Electronic Cane with ultrasonic sensors. The methodology
included a descriptive-explanatory study and the experimental design of sensors. The results
highlighted that this Electronic Cane could reduce accidents in the daily mobility of people with
visual disabilities.
In summary, notable advances have been made worldwide and nationally in creating tools and
systems to improve the lives of people with visual disabilities, using technologies such as machine
learning, computer vision, and ultrasonic sensors. These advances demonstrate the potential of
technology to increase the independence and safety of people with visual disabilities. The
research conducted in Peru emphasizes the importance of considering the specific needs of this
group and the utility of combining various methodologies to create effective solutions. Together,
these developments offer a promising outlook in which technology will continue to play a
fundamental role in improving the quality of life for people with visual disabilities, with
possibilities for application elsewhere and a focus on inclusion and autonomy.
3. Materials and Methods
The study employed an iterative, action-oriented methodology and followed the five-step process
of the Design Thinking approach (i.e., empathize, define, ideate, prototype, and evaluate) [16].
Figure 1: Design Thinking Methodology [16]
3.1. Phase 01: Empathize
This phase was rigorously executed, encompassing a comprehensive literature review of
available assistive tools and technologies. Requirements gathering was done through interviews
and analysis of local statistics, the evaluation of Artificial Vision algorithms, a detailed comparison
of IoT devices, and the meticulous selection of Computer Vision techniques. This ensured that the
device would be designed with a closer understanding of the needs and conditions of visually
impaired individuals from the Association of the Blind in Arequipa. Best practices and accessible
technologies were leveraged to provide an effective solution tailored to their primary needs,
which revolve around the recognition of banknotes and nearby everyday objects.
3.2. Phase 02: Define
After gaining an empathetic understanding of the main needs of visually impaired individuals,
the define phase involved categorizing the results obtained from the evaluation of techniques,
Artificial Vision algorithms, and IoT devices. This classification identified and selected the most
appropriate and effective approaches to address the needs of visually impaired individuals in the
study region. Additionally, a comprehensive compilation of results related to the current situation
of these individuals and available assistive tools in the context of Arequipa was conducted. These
findings provided a solid foundation for decision-making in the design of the device. It is worth
mentioning that all these results were thoroughly documented in the thesis report, thus
establishing a solid and well-founded basis for the subsequent ideation and prototyping phases.
3.3. Phase 03: Ideate
With a defined understanding of the problem, a creative flow of ideas is generated. In this
phase, a plan is designed and developed to achieve the goal of creating a product that improves
or solves the identified problem. To accomplish this, a set of actions was followed for creating the
dataset of banknote images. This included setting up the necessary image capture equipment,
involving the selection of appropriate cameras and sensors. A meticulous procedure for image
capture was developed, considering a variety of relevant scenarios and situations reflecting the
diversity of environments in which visually impaired individuals might use the device. Image
capture was carried out extensively, covering banknotes representative of the local reality.
Subsequently, labeling of these images was performed, preparing them for use in training the
Artificial Vision module. The results obtained in this stage provide a solid dataset of banknote
images, crucial for the development of the product. These achievements have been thoroughly
documented, establishing a clear and well-founded direction for the subsequent prototyping and
evaluation stages.
Table 1
Total Number of Captured Banknote Images
Denomination
Total
10 20 50 100
New Banknote Family 728 894 1190 939 3751
Old Banknote Family 1025 1162 1831 1285 5303
Total 1753 2056 3021 2224 6054
3.4. Phase 04: Prototyping
During this stage, the implementation of the vision module designed for real-time image
capture and processing, a fundamental element of the proposal, has taken place. This
achievement was not only crucial for the device's operation but also highlights the robustness of
the strategy in combining Internet of Things (IoT) technologies with advanced Computer Vision
techniques to accomplish the task of recognizing banknotes and everyday objects near the user.
Additionally, exhaustive tests have been conducted, and the components of the vision module
have been debugged. This has allowed for the effective identification and addressing of any
obstacles or potential deficiencies, ensuring optimal system performance. Furthermore,
acceptable communication has been achieved between the vision module and the IoT device,
thereby reinforcing the reliability and effectiveness of the proposed solution. The design of the
first prototype is presented below:
Figure 2: Front Design of the Prototype
Figure 3: Back Design of the Prototype
3.5. Phase 05: Evaluation
In the evaluation phase, technical and functional tests have been conducted under controlled
conditions to validate the integration of the system, measure effectiveness in banknote
recognition, and collect technical feedback. Although end-users have not yet been included in
these evaluations, crucial data has been gathered to identify areas for improvement in the design
and operation of the device. These data and observations will serve as a foundation for future
tests with end-users and contribute to the overall success of the project.
4. Results
One of the main outcomes in the development of the project proposal is:
1. The elaboration of a systematic review of assistive tools for visually impaired individuals,
conducted during the empathize stage, allowed for the identification of the main
limitations and contributions in the last 5 years of research in the field. These are related
to advances in assistive technologies, the Internet of Things, and Computer Vision. In
figures 4 and 5, the main findings of the systematic review are presented, emphasizing
the key limitations and contributions.
Figure 4: Limitations from the Systematic Review
Figure 5: Contributions from the Systematic Review
2. The creation of the dataset for Peruvian banknotes, along with its publication on the IEEE
DataPort platform. It resulted in a total of 9315 processed images, with 6568 for training,
2486 for validation, and 261 for testing. It covers a total of 16 different categories,
including both the obverse and reverse sides of 10, 20, 50, and 100 Peruvian soles from
both old and new bill families, spanning from 2011 to 2021. The images were captured
using rear cameras of mobile phones under various backgrounds and lighting conditions,
including cluttered backgrounds and images of folded banknotes.
Access link: https://ieee-dataport.org/documents/dataset-peruvian-banknotes
3. The construction of the first prototype of the proposed assistive device is a significant
outcome of the project. With this, the overall objective of providing assistance in the
recognition of Peruvian banknotes and nearby objects to the user would be achieved.
As a key contribution within the construction of the first prototype, the effective use of
pre-trained deep learning models for banknotes and models pre-trained on an ImageNet
dataset for objects, the characterization of processing complexity, the design of an
economical assistive system, and the empirical evaluation of accuracy in real-world
conditions have been highlighted. Thus, achieving the design of the first prototype, the
optimal processing of Peruvian banknote and object images, and the empirical evaluation
of the accuracy of pre-trained deep learning algorithms with 68% for banknote
recognition and 73% for object recognition with Fine-Tuning.
5. Conclusions
This study has focused on addressing the challenges faced by visually impaired individuals in
identifying banknotes in the city of Arequipa and the development of an assistive device based on
artificial vision and IoT to mitigate these challenges. From the research and work conducted, the
following conclusions can be drawn:
• The need for technological solutions for visually impaired individuals is critical. The
identification of banknotes and objects is essential in daily life, making the creation of
economical and effective assistive devices imperative to improve their quality of life.
• The exhaustive review of the state of the art in global and national banknote recognition
systems highlights that technology, particularly machine learning, computer vision, and
sensors, can be a valuable ally for visually impaired individuals, providing them with
independence and security.
• The Design Thinking approach, encompassing the stages of empathy, definition, ideation,
prototyping, and evaluation, has proven effective for the development of an assistive device
that caters to the specific needs and conditions of visually impaired individuals in Arequipa.
• The creation of a dataset of banknote images and the development of the real-time vision
module are fundamental achievements supporting the viability and effectiveness of the
proposed assistive device.
• Although tests with end-users are still pending, the technical data collected so far will
serve as a basis for future evaluations and refinements of the device. This project is anticipated
not only to enhance the quality of life for visually impaired individuals in Arequipa but also to
potentially serve as a model for similar solutions worldwide.
References
[1] C. Lizarraga, «Propuesta para el diseño de un bastón electrónico para personas invidentes
que mejorara la calidad de su desplazamiento diario,» Universidad Continental, Arequipa,
2018.
[2] C. Killa, «Pulsera para guiar a personas con discapacidad visual recibe medalla de oro en
Corea,» Agencia Peruana de Noticias, 17 Octubre 2021. [En línea]. Available:
https://andina.pe/agencia/noticia-pulsera-para-guiar-a-personas-discapacidad-visual-
recibe-medalla-oro-corea-865204.aspx. [Último acceso: 15 Septiembre 2023].
[3] N. Sufri, N. Rahmad, N. Ghazali, N. Shahar y M. Asári, «Vision Based System for Banknote
Recognition Using Different Machine Learning and Deep Learning Approach» 2019 IEEE
10th Control and System Graduate Research Colloquium (ICSGRC), pp. 5-8, 2019.
[4] S. Castelar, L. A. Banegas, D. A. Mendoza, J. C. Soto y K. Davila, «Automated Honduran
Banknote Image Classification using Machine Learning» 2022 IEEE 40th Central America and
Panama Convention (CONCAPAN), pp. 1-6, 2022.
[5] X. Huang y S. Gai, «Banknote Classification Based on Convolutional Neural Network in
Quaternion Wavelet Domain» IEEE Access, vol. 8, pp. 162141-162148, 2020.
[6] S. R. Awad, B. T. Sharef, A. M. Salih y F. L. Malallah, «Deep Learning-Based Iraqi Banknotes
Classification System for Blind People» Eastern-European Journal of Enterprise
Technologies, vol. 1, nº 115, pp. 31-38, 2022.
[7] C.-B. Fan, D. T. Aseffa, H. Kalla y S. Mishra, «Ethiopian Banknote Recognition Using
Convolutional Neural Network and Its Prototype Development Using Embedded Platform»
Journal of Sensors, vol. 2022, p. 4505089, 2022.
[8] K. Tamayo, «Sistema de Reconocimiento de Billetes para Personas con Discapacidad Visual
Mediante Visión Artificial,» Universidad EIA, Colombia, 2022.
[9] A. Khalil, M. Yaghi, T. Basmaji, M. Faizal, Z. Farhan, A. Ali y M. Ghazal, «Mobile Deep
Classification of UAE Banknotes for the Visually Challenged» 2022 9th International
Conference on Future Internet of Things and Cloud (FiCloud), pp. 321-325, 2022.
[10] E. Vela, «Diseño e implementación de un bastón ergonómico con sistema de posicionamiento
global para mejorar el desplazamiento de personas invidentes en el centro “la unión nacional
de ciegos del Perú",» Ingeniería Electrónica, Lima, 2019.
[11] M. Vilchez, «Bastón sensorial geolocalizador inteligente para apoyar en el desplazamiento de
personas invidentes en la organización regional de ciegos del Perú – Chiclayo,» Ingeniería de
Sistemas y Computación, Chiclayo, 2021.
[12] C. Lizárraga, «Propuesta para el diseño de un bastón electrónico para personas invidentes
que mejorara la calidad de su desplazamiento diario,» Ingeniería Industrial, Arequipa, 2018.
[13] H. Shah, M. Amin, K. Dadwani, N. Desai, A. Chatiwala, T. Senjyu, C. So-In y A. Joshi, «Real-Time
Object Detection System with Voice Feedback for the Blind People» Smart Trends in
Computing and Communications, pp. 683-690, 2023.
[14] Z. Yuan, T. Azzino, Y. a. L. Y. Hao, H. Pei, A. Boldini, M. Mezzavilla, M. Beheshti, M. Porfiri, T. E.
Hudson, W. Seiple, Y. Fang, S. Rangan, Y. Wang y J.-R. Rizzo, «Network-Aware 5G Edge
Computing for Object Detection: Augmenting Wearables to “See” More, Farther and Faster»
IEEE Access, vol. 10, pp. 29612-29632, 2022.
[15] M. Valipoor y A. de Antonio, «Recent trends in computer vision-driven scene understanding
for VI/blind users: a systematic mapping» Universal Access in the Information Society, 2022.
[16] P. Geerts, J. Eijsink, A. Moser, P. ter Horst, C. Boersma y M. Postma, «Rationale and
development of an e-health application to deliver patient-centered care during treatment for
recently diagnosed multiple myeloma patients: pilot study of the MM E-coach» Pilot and
Feasibility Studies, vol. 9, nº 1, 2023.
[17] N. Caytuiro-Silva, J. Peña-Alejandro y E. Castro-Gutierrez, «DATASET OF PERUVIAN
BANKNOTES,» IEEE DataPort, Arequipa, 2023.