A major political and technical challenge for a modern society is to find innovating solutions for an effective assistance to increase the security and mobility of the visually impaired persons travelling through city streets and buildings. For people who are blind, way finding is dependent on the ability to remain localized and oriented. Recent works on assistive technologies for human localization are mainly focused on GPS (Global Positioning System) (Brusnighan 1989, Makino 1997, Loomis 2001, Kowalik 2004). But GPS sensor is indoor ineffective and fails to provide accurate spatial position in low cost version (ten meters -for the best -is clearly inadequate to keep a walker along the intented path). Furthermore, GPS suffers of masking areas especially as the blind individual moves while travelling along the walls. Some works have tested differential GPS which allows around one meter accuracy (Helal 2001, Balachandran 2003), but it is a costly and cumbersome equipment which needs fixed ground stations and is only outdoor efficient. Alternative devices have also been designed for indoor human localization such as ultrasound (Ran 2004) or radio frequency identification transponder (Na, 2006), using robotic "dog-guide" (Kulyulin 2004) or instrumented white cane (Hesch 2007). To sum up, the localization systems with sufficient accuracy for pedestrian application at this time are based on ground installation (DGPS), and Drishti (Ran 2004) is the only system which proposes a combined outdoor/indoor system. To trust in a large spread, the design of an individual localization system will be based on wearable, low cost and mobile technologies, free from collective equipment. In this way, we propose a new vision based solution derived from autonomous navigation of wheeled robot (Royer 2005) to assist visually impaired in their everyday travelling outdoor as well as indoor. This computer vision based system requires no environment instrumentation: instantaneous accurate localization and heading estimates of the user are computed only from images provided by a body mounted camera as the trip progresses along a memorized path. This real time localization is available provided that a video sequence has been previously recorded during a learning trip. For the time being, the vision system is only used to localize and guide the walker along the intended path, it does not substitute for the long white cane in obstacle avoidance. The following section describes our vision based positioning system. Some experimental results with a first wearable prototype are given in section 3. The conclusion and future works are discussed in the last section.
The localization system relies on two steps (Figure 1): a preliminary learning path and a real time localization for navigation assistance. During a learning trip, a video sequence is recorded by the onboard vision system and processed off-line to automatically build a large set of natural landmarks by applying structure from motion algorithm. This algorithm permits to jointly recover the motion of the camera and the geometry of a rigid surrounding scene by automatically recognizing some singular points through the image sequence (angular points on buildings, windows, doors, billboard, trees,…). The recorded reference consists in a set of selected key images corresponding to different camera positions along the learning path and a 3D map of the environnement. During navigation assistance, ego motion of the camera is computed as the trip progresses by matching the stored 3D landmarks with their projections extracted in the current image. Figure 2 shows a result of localization during a trip in the university library. As the camera is rigidly allied to the carrier, it provides its instantaneous localization and orientation (6 DoF) along the trip. Even if transient obstacles (pedestrian, car, advertising hoarding), that were not in the path of the memorized route, mask some part of the 3D environment, the camera localization may be effective while there are at least a few tens of well recognized landmarks. Figure 2 presents such a case with pedestrian shifting. Initial position is automatically detected by scanning the stored key images while camera is placed not too far from the memorized path. Performance of the system has been largely tested with wheeled robots: absolute localization accuracy around 0.05m is achieved by comparing with the ground truth given by a DGPS sensor (Royer 2005). With such a low cost vision system, the robot may be controlled with the same accuracy that with the DGPS receiver. We propose to adapt this system to pedestrian navigation by using a body mounted camera and a wearable computer. The vision system provides current localization and also heading of the walker relative to the local segment of the reference path. Considering this current position, the walker may be continuously reoriented to be kept in a navigation corridor along the intended path by sound effects or vocal prompts.
Figure 1 Synoptic of the vision based localization system (left) and the pedestrian prototype with a chest mounted camera and a wearable computer in a back pack (right).
S. Treuillet, E. Royer, T. Chateau, M. Dhome, J.M. Lavest
A first pedestrian prototype has been developped to test the robustness of the vision based localization algorithm on bumpy walking trips (unlike the wheeled robots which move smoothly). Several video sequences have been acquired along walking trips more than 100m long in various situations (indoor/outdoor, open/cluttered space, flat floor/stairs,…). Different inexperienced visually impaired subjects test the localization system several days or weeks after the video sequence acquisition of the learning trip. The navigation performance may be analyzed by observing in live the path followed by the walker in comparison with the intended one (i.e. the reference path followed by the camera during the learning trip).
Figure 2 Left: Top view of a walking trip inside the university library. Empty squares represent all the camera positions selected during the learning path. The current position and heading estimates of the person over the intended path (colored square and vector), bounded with uncertainty ellipsoïd on camera position estimates (extented 100 times). Right: Current image captured by the camera (bottom) and the corresponding image-key memorized during the learning trip several days before (top). Large colored squares are the well matched points, small ones the unmatched.
A photo of the experimental device is given in Figure 1. The pedestrian prototype is composed of a 2,8mm lens equipped AVT Guppy 044B camera (120° diagonal field of view) fixed on a body harness and connected to a wearable computer (Pentium M 1.86GHz 1 Go RAM). Intrinsic parameters of the camera including radial distortions are estimated by a prior calibration. The image resolution is 320x240 pixels which allows to provide localization data every 40 ms. From these localization and heading measurements, vocal prompts are given to keep the walker as close as possible to the memorized path. In this primary version, two different guiding modalities have been tested with headphones: a speech-based interface or sonar sound effect regularly time spaced. The verbal navigational prompts are of three types: "turn left", "turn right" or "straight ahead" depending on walker's heading. Similarly, the sonar-ping are respectively applied only in the left ear, only in the right ear or in both, with varying frequency according to the adherence to the path (angular deviation). S. Treuillet, E. Royer, T. Chateau, M. Dhome, J.M. Lavest
In order to illustrate the efficiency of the proposed vision system, Figure 3 shows the tracks of three subjects made blind by an opaque bandage on their eyes navigated outdoor. Equipped with our vision based localization system, they have been able to walk near the reference path in unfamiliar open space without point of reference. Table 1 gives some statistics on navigation performance during this outdoor trip and the indoor trip inside the library whose a snapshot is given in Figure 2. The average lateral space relative to the reference path is around 0.50m without optimization or subject apprenticeship. Guidance system integrates no motion model and except the prior camera calibration, no harness calibration has been made when exchanging participants. This proves the relative robustness of the system to variations of acquisition conditions. 4 along a totally open space indoor trip without wall to follow. These experimental results demonstrate the effectiveness of the vision based localization system to keep the walker in a navigation corridor less than one meter width along the intended path.
A wearable vision system has been developed for completely localizing a person on a memorized path. Vision based localization allows the same accuracy that the DGPS receiver without the drawbacks of this cumbersome and costly system. Our real-time localization system may be used for pedestrian navigation both outdoor and indoor without dependences to the environment instrumentation. It is clearly an advantage to design a wide used autonomous system for assisting visually impaired people. Future works are dedicated to increase the ergonomics and robustness of the system by working in collaboration with end-users. The key issue to improve guidance efficiency is to find the best communication modalities taking account the psychological and ergonomic factors with a large panel of users with profound vision loss in various situations. Navigation control may be analysed in live by the quantitative feedbacks of path efficiency which will be a very usefull tool to investigate different multimodal adaptative interfaces. We have observed that performance improves with practice. Machine needs to learn to work with the user and adapt some walk models to each user. We also plan to investigate a second prototype mixing a glasses mounted camera with 3-axis gyroscopes to estimate the movements of the head and developing application on PDAs. The ambition of the project is to lead to a convenient embedded personal guidance system dedicated to visual impaired users in complement of existing obstacle detection device like long cane. Such a system may also be useful for locomotion learning of blind people or for elderly people with memory loss to improve their mobility and social inclusion.
The authors thank Gregory Gérenton for its support in the software optimisation and his active contribution to experiments during his training course in laboratory. Special thanks are also due to Nièle Pouzet, a blind student of the University, for her disposal to test our prototype.