One of the challenges faced by blind people is the difficulty in identifying objects with concise information. They could only rely on the senses of hearing, smell, taste or touch to engage and get some perspectives of objects. Hence, this paper presents a mobile application called Iris to aid blind people in “visualising” their surroundings with descriptive objects. Iris combines the multiple object detection and optical character recognition capabilities of Microsoft Computer Vision API to turns smartphones into assistive devices for the blind to use in their daily activities.
According to a study in 2010, visual impairment is a major
global health issue affecting 285 million people in six
World Health Organization regions. Approximately 39
million of them are blind, and 246 million with decreased
visual acuity
Among the issues of current mobile applications for blind people is it could only detect a single object at a single frame. In a real-life environment, there is a tendency for objects to be close to each other. Therefore, it is crucial to develop a mobile application that could detect not only one but multiple objects in a single camera frame. Another issue is that most applications (e.g. BlindTool and Aipoly) respond without additional contexts or descriptions of objects with their surrounding environment. For instance, if an apple is detected, the application speaks out the word “apple” to the user. Little does the user know, the apple may be on top of a table. Moreover, printed text is everywhere in our daily life, such as on reports, receipts, bank statements, product packages and medicine bottles. It is troublesome for blind people if they are unable to read these texts. Therefore, this paper developed a mobile application called Iris for the utility of blind people. The user can tap on the screen to detect descriptive objects. A captured image is sent to Microsoft Computer Vision API as a parameter to retrieve values of detected objects and transcribed text in the image. The mobile application then sorts the retrieved object descriptions and text transcriptions based on confidence to form a sentence. Finally, the application speaks out the sentence to the user in a natural language.
The rest of the paper is organised as follows. First, section 2 gives an overview and discusses some of the related works. Section 3 provides an overview of Iris’s design and its main components. Then, the proposed Iris’s design is presented in section 4. Subsequently, section 5 concludes this paper. 2
In 1996, Malaysia’s National Eye survey found that among
18,027 residents examined, the age-adjusted prevalence of
blindness and low vision were 0.29% and 2.44%
respectively. Females had a higher age-adjusted prevalence of low
vision compared to males
With higher computational and storage capacity of mobile
devices, as well as growing speeds and coverage of mobile
internet, provide unique possibilities for the use of
smartphones as universal assistive devices
This study had benefitted from the use of existing
accessibilities technologies. People with visual impairment can
easily browse and navigate through their smartphones with
accessibility features to use the proposed mobile
application. Table 1 shows some of the accessibility features for
blind and low vision people, which are available in mobile
devices using iOS
iOS VoiceOver Speak Screen Captioning & audio descriptions Dark mode and smart invert colours Zoom and font adjustment Magnifier Accessibility Shortcuts Dictation Braille entry & display
Android TalkBack Select to Speak Audio & on-screen text Contrast and colour options Change display and font size Magnification Interaction controls Voice dictation
BrailleBack
Object recognition brings forth a multitude of possibilities
in the modern world. This study had also implemented the
use of multiple object detection technology. An object
detection algorithm typically creates a bounding box around
the object of interest to locate it within the image. However,
the algorithm might not necessarily draw just one bounding
box in an object detection case, there could be many
bounding boxes representing different objects of interest within
the image, and it would not know how many beforehand
Faster R-CNN (Region Convolutional Neural Network) was
developed by researchers at Microsoft, which is based on
RCNN with a multi-phased approach to object detection.
RCNN used selective search to determine region proposals,
pushed these through a classification network and then used
a Support Vector Machine (SVM) to classify the different
regions
Requirement analysis is done to understand what will be built, why it should be built, and in what order it should be built. This section explains in detail about the needs of the target users of this application, which are blind people. Interviews were conducted with three voluntary respondents, courtesy of Malaysian Association for the Blind in Kuala Lumpur. Two of them are males, and one is female. They are aged between 24 and 31 years old. The purpose of the interviews was to understand the challenges that blind people face in identifying objects, to get their perception regarding mobile applications, and get their inputs on the development of Iris. The following subsections discussed the results of the interviews. 3.1
Most of them have problems identifying new objects even by touching them. It means that when a product that they have been buying had changed the packaging, then they would have difficulty to identify the product without someone telling them what it is. Problems also arise when they are unable to touch something, and they could not determine what objects are present in their proximity, or even the environment or location they are situated in. They would need to depend on other senses, such as sound and smell, which could be tricky if it is a new environment. The respondents agreed that they prefer to know what environment they are in without having to touch around.
Besides, the respondents have trouble distinguishing two physically similar objects. They are not able to tell one object from the other when the objects have the same textures or properties when touched. Hence, they need the ability to differentiate objects. For example, they might want to differentiate between a can of tomato soup from a can of evaporated milk. This problem also relates to their inability to read texts if they are not in braille. 3.2
All of the respondents have their personal smartphones. However, their primary uses of a smartphone merely to text and call. They use assistance features that come with their phones to navigate and interact with the phone’s interfaces. The respondents did not use any assistance applications with their smartphones, but they surely welcome any applications that could support their visual needs. One of the respondents shared that they wanted a mobile application that could tell if their hair or cloth is messy. Another respondent shared that he wanted an application that would tell him if there are things in front of him and to tell him the description of certain items. The responded also added that he wanted an application to manage his medications because it could be confusing sometime.
Due to usability reasons, all of the respondents agreed that tapping the screen would be the best way to take a picture. It presumably because it is easier to just tap anywhere on the screen rather than having to locate a button. One key feature suggested by a respondent was an auto-flash feature. This feature proves to be useful to blind people because they would not know if the scene is dark. Hence, having the autoflash feature helps with the overall usability of the application. Another key feature suggested was repeating the application’s instructions. This feature was suggested to allow repetition of instructions for new users and without having to go through the process of retaking the picture. This part of the interview had helped in designing the application to be more user-friendly to blind people. 4 4.1
After uploading an image, Computer Vision API’s output tags based on the objects, living beings, and actions identified in the image. Tagging is not limited to the main subject, such as a person in the foreground, but also includes the setting (indoor or outdoor), furniture, tools, plants, animals, accessories, gadgets and others. Computer Vision API’s algorithms analyse the content in an image. This analysis forms the foundation for a “descriptive” displayed in complete sentences. Computer Vision API’s algorithms generate various descriptions based on the objects identified in the image. The descriptions are each evaluated and a confidence score generated. An ordered list is then generated from the highest confidence score to the lowest.
OCR technology detects text content in an image and
extracts the identified text into a machine-readable character
stream
is also spoken to the user. As previously discussed, in this specific case, the API had not been able to return a result in descriptions. Hence, the tags from the result’s JSON are accessed and mapped; objects to their characteristics. It makes it possible for the application to output a result even when the Computer Vision API could not construct a description from the picture.
Figure 4 (a) shows an output when there are texts associated with the object detected. It can be seen that the outputs from two of the APIs used are combined within the application to create an output that is intuitive to the user. While Figure 4 (b) shows an exception case when Iris is opened in a poorlylit environment. The figure depicts that the light source is coming from the smartphone’s flashlight. In this case, the application detected that the ambience value in the darkroom is too low. To counter that issue, Iris automatically toggles the smartphone’s flashlight on. With this feature, it is more accessible to blind people in the sense that they could not tell if they are in a dark environment while using Iris. Therefore, it is a feature that helps illuminate objects while being in the dark. In a nutshell, this paper adopted multiple object recognition and OCR technologies to develop an assistive mobile application for blind people. Iris helps to identify objects and produce descriptive texts from a picture taken by a camera. It serves to describe and distinguish objects so that the blind can have a better insight into the objects in their surroundings. The overall outcome was satisfactory, considering that blind people can make use of the proposed mobile application to tackle problems in their daily activities, hence, aiding them towards independence. However, there are plenty of features that could be implemented to improve the application, such as lower latency and face detection. With 5G technology a major availability in the future, the application could access advanced algorithms and image processing services in the cloud and retrieve the results almost instantaneously. Moreover, the application could be updated to detect and describe human faces, which will be a great support in the communication of blind people with others.