=Paper=
{{Paper
|id=Vol-415/paper-4
|storemode=property
|title=Body mounted vision system for visually impaired outdoor and indoor wayfinding assistance
|pdfUrl=https://ceur-ws.org/Vol-415/paper4.pdf
|volume=Vol-415
|dblpUrl=https://dblp.org/rec/conf/cvhi/TreuilletRCDL07
}}
==Body mounted vision system for visually impaired outdoor and indoor wayfinding assistance==
https://ceur-ws.org/Vol-415/paper4.pdf
Conference & Workshop on Assistive Technologies for People with Vision & Hearing Impairments
Assistive Technology for All Ages
CVHI 2007, M.A. Hersh (ed.)
BODY MOUNTED VISION SYSTEM FOR VISUALLY IMPAIRED
OUTDOOR AND INDOOR WAYFINDING ASSISTANCE
Sylvie Treuillet, Eric Royer, Thierry Chateau, Michel Dhome, Jean-Marc Lavest
LASMEA – UMR 6602
Campus des Cézeaux, 63177 Aubière, France
Phone: 33 473 407 764, Email: Sylvie.treuillet@univ-bpclermont.fr
Abstract: The most challenging issue of the navigation assistive systems for the visually impaired is
the instantaneous and accurate spatial localization of the user. Most of the previous proposed systems
based on GPS sensors have clearly insufficient accuracy for pedestrian use and are confined to
outdoor with severe failing in urban area. This paper presents an interesting alternative localization
algorithm using a body mounted single camera. Instantaneous accurate localization and heading
estimates of the person are computed from images as the trip progresses along a memorised path. A
first portable prototype has been tested for outdoor as well as indoor pedestrian trips. Experimental
results demonstrate the effectiveness of the vision based localization to keep the walker in a
navigation corridor less than one meter width along the intended path. Future works will investigate
multimodal adaptative interface taking account the psychological and ergonomic factors for the blind
end-user to design a suitable guiding solution for the blind and visually impaired.
Keywords: personal localisation and heading system, navigation assistance, blind people
1. Introduction
A major political and technical challenge for a modern society is to find innovating solutions for an
effective assistance to increase the security and mobility of the visually impaired persons travelling
through city streets and buildings. For people who are blind, way finding is dependent on the ability to
remain localized and oriented. Recent works on assistive technologies for human localization are
mainly focused on GPS (Global Positioning System) (Brusnighan 1989, Makino 1997, Loomis 2001,
Kowalik 2004). But GPS sensor is indoor ineffective and fails to provide accurate spatial position in
low cost version (ten meters - for the best - is clearly inadequate to keep a walker along the intented
path). Furthermore, GPS suffers of masking areas especially as the blind individual moves while
travelling along the walls. Some works have tested differential GPS which allows around one meter
accuracy (Helal 2001, Balachandran 2003), but it is a costly and cumbersome equipment which needs
fixed ground stations and is only outdoor efficient. Alternative devices have also been designed for
indoor human localization such as ultrasound (Ran 2004) or radio frequency identification transponder
(Na, 2006), using robotic “dog-guide” (Kulyulin 2004) or instrumented white cane (Hesch 2007). To
sum up, the localization systems with sufficient accuracy for pedestrian application at this time are
based on ground installation (DGPS), and Drishti (Ran 2004) is the only system which proposes a
combined outdoor/indoor system. To trust in a large spread, the design of an individual localization
system will be based on wearable, low cost and mobile technologies, free from collective equipment.
In this way, we propose a new vision based solution derived from autonomous navigation of wheeled
robot (Royer 2005) to assist visually impaired in their everyday travelling outdoor as well as indoor.
This computer vision based system requires no environment instrumentation: instantaneous accurate
localization and heading estimates of the user are computed only from images provided by a body
S. Treuillet, E. Royer, T. Chateau, M. Dhome, J.M. Lavest
mounted camera as the trip progresses along a memorized path. This real time localization is
available provided that a video sequence has been previously recorded during a learning trip. For the
time being, the vision system is only used to localize and guide the walker along the intended path, it
does not substitute for the long white cane in obstacle avoidance. The following section describes our
vision based positioning system. Some experimental results with a first wearable prototype are given
in section 3. The conclusion and future works are discussed in the last section.
2. Vision Based Positioning
The localization system relies on two steps (Figure 1): a preliminary learning path and a real time
localization for navigation assistance. During a learning trip, a video sequence is recorded by the on-
board vision system and processed off-line to automatically build a large set of natural landmarks by
applying structure from motion algorithm. This algorithm permits to jointly recover the motion of the
camera and the geometry of a rigid surrounding scene by automatically recognizing some singular
points through the image sequence (angular points on buildings, windows, doors, billboard, trees,…).
The recorded reference consists in a set of selected key images corresponding to different camera
positions along the learning path and a 3D map of the environnement. During navigation assistance,
ego motion of the camera is computed as the trip progresses by matching the stored 3D landmarks
with their projections extracted in the current image. Figure 2 shows a result of localization during a
trip in the university library. As the camera is rigidly allied to the carrier, it provides its instantaneous
localization and orientation (6 DoF) along the trip. Even if transient obstacles (pedestrian, car,
advertising hoarding), that were not in the path of the memorized route, mask some part of the 3D
environment, the camera localization may be effective while there are at least a few tens of well
recognized landmarks. Figure 2 presents such a case with pedestrian shifting. Initial position is
automatically detected by scanning the stored key images while camera is placed not too far from the
memorized path. Performance of the system has been largely tested with wheeled robots: absolute
localization accuracy around 0.05m is achieved by comparing with the ground truth given by a DGPS
sensor (Royer 2005). With such a low cost vision system, the robot may be controlled with the same
accuracy that with the DGPS receiver. We propose to adapt this system to pedestrian navigation by
using a body mounted camera and a wearable computer. The vision system provides current
localization and also heading of the walker relative to the local segment of the reference path.
Considering this current position, the walker may be continuously reoriented to be kept in a navigation
corridor along the intended path by sound effects or vocal prompts.
Figure 1 Synoptic of the vision based localization system (left) and the pedestrian prototype with a
chest mounted camera and a wearable computer in a back pack (right).
2
S. Treuillet, E. Royer, T. Chateau, M. Dhome, J.M. Lavest
3. Experiments
A first pedestrian prototype has been developped to test the robustness of the vision based
localization algorithm on bumpy walking trips (unlike the wheeled robots which move smoothly).
Several video sequences have been acquired along walking trips more than 100m long in various
situations (indoor/outdoor, open/cluttered space, flat floor/stairs,…). Different inexperienced visually
impaired subjects test the localization system several days or weeks after the video sequence
acquisition of the learning trip. The navigation performance may be analyzed by observing in live the
path followed by the walker in comparison with the intended one (i.e. the reference path followed by
the camera during the learning trip).
Figure 2 Left: Top view of a walking trip inside the university library. Empty squares represent
all the camera positions selected during the learning path. The current position and heading estimates
of the person over the intended path (colored square and vector), bounded with uncertainty ellipsoïd
on camera position estimates (extented 100 times). Right: Current image captured by the camera
(bottom) and the corresponding image-key memorized during the learning trip several days before
(top). Large colored squares are the well matched points, small ones the unmatched.
3.1 Experimental device
A photo of the experimental device is given in Figure 1. The pedestrian prototype is composed of a
2,8mm lens equipped AVT Guppy 044B camera (120° diagonal field of view) fixed on a body harness
and connected to a wearable computer (Pentium M 1.86GHz 1 Go RAM). Intrinsic parameters of the
camera including radial distortions are estimated by a prior calibration. The image resolution is
320x240 pixels which allows to provide localization data every 40 ms. From these localization and
heading measurements, vocal prompts are given to keep the walker as close as possible to the
memorized path. In this primary version, two different guiding modalities have been tested with
headphones: a speech-based interface or sonar sound effect regularly time spaced. The verbal
navigational prompts are of three types: “turn left”, “turn right” or ”straight ahead” depending on
walker’s heading. Similarly, the sonar-ping are respectively applied only in the left ear, only in the right
ear or in both, with varying frequency according to the adherence to the path (angular deviation).
3
S. Treuillet, E. Royer, T. Chateau, M. Dhome, J.M. Lavest
3.2 Results
In order to illustrate the efficiency of the proposed vision system, Figure 3 shows the tracks of three
subjects made blind by an opaque bandage on their eyes navigated outdoor. Equipped with our vision
based localization system, they have been able to walk near the reference path in unfamiliar open
space without point of reference. Table 1 gives some statistics on navigation performance during this
outdoor trip and the indoor trip inside the library whose a snapshot is given in Figure 2. The average
lateral space relative to the reference path is around 0.50m without optimization or subject
apprenticeship. Guidance system integrates no motion model and except the prior camera calibration,
no harness calibration has been made when exchanging participants. This proves the relative
robustness of the system to variations of acquisition conditions.
Figure 3 Left: Outdoors tracks of three visually impaired subjects equipped with our vision
based localization system in unfamiliar open space environment. Right: A surrounding view of the
scene.
Table 1 Indoor and outdoor localization and navigation performance.
Outdoor trip (150 m) Subject 1 Subject 2 Subject 3
Average of lateral space (m) 0.59 0.34 0.66
Standard deviation of lateral space (m) 0.35 0.20 0.43
Indoor trip (70 m) Subject 1 Subject 2 Subject 3
Average of lateral space (m) 0.36 0.40 0.47
Standard deviation of lateral space (m) 0.25 0.22 0.27
A comparison of the tracks obtained with different guiding modalities is illustrated in Figure 4 along a
totally open space indoor trip without wall to follow. These experimental results demonstrate the
effectiveness of the vision based localization system to keep the walker in a navigation corridor less
than one meter width along the intended path.
4
S. Treuillet, E. Royer, T. Chateau, M. Dhome, J.M. Lavest
4. Conclusions and Future Work
A wearable vision system has been developed for completely localizing a person on a memorized
path. Vision based localization allows the same accuracy that the DGPS receiver without the
drawbacks of this cumbersome and costly system. Our real-time localization system may be used for
pedestrian navigation both outdoor and indoor without dependences to the environment
instrumentation. It is clearly an advantage to design a wide used autonomous system for assisting
visually impaired people. Future works are dedicated to increase the ergonomics and robustness of
the system by working in collaboration with end-users. The key issue to improve guidance efficiency is
to find the best communication modalities taking account the psychological and ergonomic factors with
a large panel of users with profound vision loss in various situations. Navigation control may be
analysed in live by the quantitative feedbacks of path efficiency which will be a very usefull tool to
investigate different multimodal adaptative interfaces. We have observed that performance improves
with practice. Machine needs to learn to work with the user and adapt some walk models to each user.
We also plan to investigate a second prototype mixing a glasses mounted camera with 3-axis
gyroscopes to estimate the movements of the head and developing application on PDAs. The
ambition of the project is to lead to a convenient embedded personal guidance system dedicated to
visual impaired users in complement of existing obstacle detection device like long cane. Such a
system may also be useful for locomotion learning of blind people or for elderly people with memory
loss to improve their mobility and social inclusion.
Figure 4 Two successive tracks of a same blind subject guided by the verbal navigational
prompts (vocal), respectively by the sonar interface (bip) with frequency according to the angular
deviation.
Acknowledgement
The authors thank Gregory Gérenton for its support in the software optimisation and his active
contribution to experiments during his training course in laboratory. Special thanks are also due to
Nièle Pouzet, a blind student of the University, for her disposal to test our prototype.
5
S. Treuillet, E. Royer, T. Chateau, M. Dhome, J.M. Lavest
References
Brusnighan, D.A. et al. (1989). Orientation aid implementing the global positioning system,15th IEEE
Annual Northeast Bioengineering Conference, pp.33-34.
Balachandran, W. et al. (2003). A GPS-based navigation aid for the blind, Int. IEEE Symposium on
Wearable Computers, pp.34-36.
Helal, A. et al. (2001). Dristi: an integrated navigation system for the visually impaired, Int. IEEE
Symposium on Wearable Computers, pp.149-156.
Hesch J.A. and S.I. Roumeliotis (2007). An Indoor localization aid for the visually impaired, IEEE Int.
Conf. on Robotics and Automation, Roma, Italy, April 10-14, pp.3545-3551.
Kowalik, R. and S. Kwasniewski (2004). Navigator – a talking GPS receiver for the blind, 9th Int. Conf.
on Computers Helping People with Special Needs, Paris, France, July 7-9, Lecture Notes in
Computer Science, Vol.3118, Springer Berlin, pp.446-449.
Kulyukin, V., C. Gharpure, J. Nicholson and S. Pavithran (2004). RFID in robot assisted indoor
navigation for the visually impaired, IEEE Int. Conf. on Intelligent Robats and Systems, Sandaï,
Japan, Sept. 28 – Oct. 2, vol. 2, pp.1979-1984.
Loomis, J. et al. (2001). GPS-based navigation systems for the visually impaired, Fundamentals of
Wearable Computers and Augmented Reality, pp.429-446.
Makino, H. et al. (1997). Development of navigation system for the blind using GPS and mobile phone
combination, IEEE Int. Conf. of Engineering in Medicine and Biology, vol.2, pp.506-507.
Na, J. (2006). The blind interactive guide system using RFID-based indoor positioning system, 10th
Int. Conf. on Computers Helping People with Special Needs, Linz, Austria, July 11-13, Lecture
Notes in Computer Science, Vol.4061, Springer Berlin, pp.1298-1305.
Ran, L., S. Helal and S. Moore (2004). Drishti: an integrated indoor/outdoor blind navigation system
and service, 2nd Annual Conf. on Pervasive Computing and Communications, March 14-17,
Orlando, USA, pp.23-32.
Royer, E., M. Lhuillier, M. Dhome and T. Chateau (2005). Localization in urban environments:
monocular vision compared to a differential GPS sensor, Int. Conf. on Computer Vision and Pattern
Recognition, June 20-25, San Diego, USA, vol.2, pp.114-121.
6