In this experiment, we test the hypothesis that a 'retina-like' space variant sampling pattern can improve the efficiency of a visual prosthesis. Subjects wearing a visuo-auditory substitution system were tested for their ability to point at visual targets. The test group (space-variant sampling), performed significantly better than the control group (uniform sampling). The pointing accuracy was enhanced, as was the speed to find the target. Surprisingly, the time spanned to complete the training was also reduced, suggesting that this space-variant sampling scheme facilitates the mastering of sensorimotor contingencies.
Visual prosthesis are devices that interface a video-camera with the brain at different levels : either
directly implanted on the retina or on the cortex surface
The main difference between visual prosthesis and natural vision systems is likely to be the number of
stimulation points available. As compared to the 6 million cones in the human eye or the 80000 pixels
of a video camera, visual prosthesis resolution, all categories considered, ranges from 64
In this article, we show how recent advances into the comprehension of visual perception in terms of
sensorimotor contingencies
To address the question of space-variant sampling, we first needed to choose among all possible
sampling distribution laws. The law we choose is directly inspired from observation on primates visual
system. As described by
ρ' = log(ρ + a)
Biomimeticity and focus/context strategy are not the only arguments in favor of a logarithmic mapping of the visual eccentricity: it may also bring new regularities in sensori-motor coupling. Let us compare the properties of a logarithmic cortical mapping with respect to a linear one. An object of size dX is positioned at the eccentricity X in a visual plane at a distance Z from the observer (fig. 1). It's correlate is an object of size dY at the eccentricity Y in the cortex. The central lens symbolizes the eye. For
Z Assuming a logarithmic cortical mapping, eq. 1 becomes simplification purpose, the distance between the central lens and the cortex plane is chosen as unit distance. With linear mapping, we obtain (c index stands for logarithmic coordinates):
X X ⎛ dX dZ ⎞ Y = (1) which, by differentiation, gives dY = ⎜ − ⎟ (2)
Z ⎝ X Z ⎠ ⎛ X ⎞ ⎛ dX dZ ⎞ Yc = log ⎜ ⎟ (3) which, by differentiation, gives dYc = ⎜ − ⎟ (4),
⎝ Z ⎠ ⎝ X Z ⎠ Formula (4) brings new regularities in the link between the visual world and its cortical correlate with respect to sensori-motor coupling, particularly in the case of a motion along the direction Z: 1- An object O contained in the vertical plane (dZ=0) has a constant size on the cortex regardless of the viewing distance.
⎛ dX ⎞ Indeed, dZ=0 in eq 3 gives dYc = ⎜ ⎟ (5). As dX and X are constant, dYc is also constant.
⎝ X ⎠ 2− When approaching at a constant velocity v toward an object O contained in the vertical plane (dX/dt=0), the velocity of its projection on the cortical plane is inversely proportionnal to the time to contact (T) between the object and the observer.
Indeed, temporal derivation of eq. 3 with dX/dt=0 gives Thus, logarithmic mapping of the visual world simplifies the relationship between the subject's motion and the changes it implies in his sensations. It is the reason why we claim that it brings new sensorimotor regularities. Our hypothesis is that the use of such a mapping in a visual prosthesis should enhance its efficiency. dt dYc 1 dZ 1 = − =
Z dt
T (6).
The mapping function we use is a Michaelis-Menten law which is linear for small eccentricities, logarithmic for medium ones and then saturates. One of its major advantages with respect to a logarithmic law is that it is bounded, thus fitting with the finite cortical space. With ρ being the eccentricity of a point in the visual space and ρ' its correlate in the cortical space, this law can be written :
ρ ρ ′ = ρl′im ρ + ρ o (7) ρ'lim and ρ0 are determined so that the image size is preserved and that the central region magnification factor (lately referred to as R0) is adjustable. The central magnification factor R0 is define as the ratio between sampling density at the excentricity ρ=0 over global sampling density.
Figure 2 : Mapping of the cortical eccentricity ρ' as a function of the visual eccentricity ρ.
TheVibe device is an experimental system for the conversion of images into sound patterns
To design standard uniform retinas for TheVibe, we cut the 320 x 240 image into a set of 16 x 12 cells, each side 20x20 pixels. A receptive field origin is chosen at a random position in each cells. Each receptive field is composed of 10 sampling pixels chosen randomly in a 20 x 20 box centered on the receptive field origin so that overlap with other receptive field is possible. Our retinas were composed by 192 receptive fields.
Frequency and inter-aural disparity followed a linear mapping of the vertical (resp. horizontal) position.
Frequencies ranged from 300 to 3000 Hz. To design space-variant retinas, a logarithmic mapping as defined Fig 3 : A space-variant logarithmic (R0=2) in section 2, with a central magnification factor R0=2, retina for TheVibe. Note that the extent of was applied to uniform retinas. The result is illustrated in the receptive fields increases with fig 3. Initial linear mapping in the auditory space was eccentricity. In particular, our mapping kept. preserves overlap.
The test protocol was inspired by the work of
The subject was equiped with a wide angle webcam (Logitech, Quickcam Pro 5000) at the top of his
head, connected to a Dell GX620 with two 3Ghz Pentium 4 processors PC. Auditory feedback was
transmitted via Sennheiser HD 280 headphones. The subject was placed in front of a screen were a
white target, 8° in diameter was presented on a black background at different locations (fig. 4). The
projected picture covers exactly the 78° x 58° field of view of the substitution device. Targets were
generated with PsychToolbox
The experiment was separated in three stages divided by 5 mn breaks.The first stage was “free exploration”. For 5 minutes, the subject, standing in front of a unique target, freely explored it's visual field. He was allowed to move his head and body as he wished. The next 5 minutes, subject was asked to move to the right and to the left while keeping the target fixed with respect to the vision device, first at ½ m of the screen and then at 1m. The second stage was “masking”. In this task, the subject was seated in front of the screen. For the first thirty randomized target positions, he had to point to the target with his head and then to mask it with his hand, keeping his arm extended (so he does not mask the whole camera aperture). For the next thirty trials, after a 5 mn break, he had to mask the target with his hand Fig 4: Experimental without pointing at it with the vision device. Duration of the whole second room and setup stage was recorded. The last stage was the test. Forty targets were presented on the screen at 20 different positions on the subject's visual field (each one was presented twice). The subject, seated in front of the screen, was requested to “place the target at the center of his perceptive field, as accurately and quickly as possible”. He had to validate when he felt it was the case. Position of the target in the visual space and time spent for each trial were recorded.
Fourteen subjects, most of them students (m = 26 y, σ = 4 y) took part in the experiment; none of them had ever used this type of device. Seven were in the space-variant condition, seven in the uniform condition. Groups were paired for age and for gender. Results are described in fig 5.
Accuracy
Response time
Training duration Target localization: we computed the mean position for all trials, which we called the “subjective center”. We then computed the distance between this subjective center and the actual position of the target at the end of each trial which is the data we used for assessing accuracy. Our hypothesis was that the accuracy should be enhanced when using a space-variant sampling. A unilateral Mann & Whitney test was applied to address this question. Subjects in space-variant condition (Log2) performed significantly better (z = 4,99 ; puni< 0.001) with mean mlog2 = 35 pix. and standard deviation σlog2 = 29, against munif = 44 pix., σunif = 28 in the uniform condition.
Time to focus the target: the mean time required to focus at a visual target in test stage has been measured to test the efficiency of the device. Subjects in space-variant condition performed significantly better (Unilateral Mann & Whitney test, z = 6,96 ; puni< 0.001) with mean mlog2 = 10,58 s and standard deviation σlog2 = 6,68 s, against munif = 19 ,46 s, σunif = 17,47s in the uniform condition. Training duration: the duration of the second stage of training was also measured in order to assess difficulty in the device appropriation. As we had no idea about the direction of this effect, we performed a bilateral Mann & Whitney test. Subjects in space-variant condition performed significantly better (U= 47,5; pbil< 0.01) with mean mlog2 = 23,4 min and standard deviation σlog2 = 4,6min, against munif = 36,3 min, σunif = 6,7 min in the uniform condition.
Thus, with a shorter learning time and shorter response time, subjects in the space variant condition performed significantly better in terms of accuracy.
Although quite promising, these results need to be carefully considered, for two primary reasons. When applied to a rectangular image, our mapping induces blank zones that are not sampled (at the borders). The proportion of this effect is related to the magnification factor. For R0=2, blank zones are approximately 10% of the picture. Although this aspect is not likely to explain our results, its effect can not be excluded. We are currently designing circular retinas that will not endorse this effect. Another possible caveat is the fact that the operator was aware of the subject's group when conducting the experiment. Even though the operator could not have had direct influence on the measures, results need to be confirmed with a double blind protocol.
However, this study shows that new pathways exist which can possibly enhance vision prosthesis, even though their resolution is bound to be limited. It provides an experimental framework to address their practical performances and it shows that significant and even counter-intuitive effects may be obtained. The fact that accuracy would be enhanced was natural since the sampling network was denser at the foveal center. Shorter response time may also be understood as an effect of the sampling variation, which provided additional information to locate a position in the visual field. On the other hand, space-variant sampling could have complicated the mastering of the vision device: it appears to be the opposite. This result leads to the idea that our mapping may facilitate the mastering of sensori-motor laws. This last aspect needs to be addressed further to determine whether our mapping is optimal or whether other kinds of space-variant sampling, a linearly decreasing distribution law for instance, may have the same effect.
Lastly, one may take a practical view of this study. To date, vision devices are far from giving blind people new eyes. However, contrary to surgical approaches (cataract operations, retinal transplant), electronic vision devices may easily be adapted to specific tasks. By giving the subject a better ability to locate a visual target, space variant sampling may be of use in devices designed for orientation, object finding and mobility assistance. Acknowledgments: The “Sensory Substitution System for Visual Handicap” project is granted by the Rhône-Alpes region and the association “Les Gueules Cassées”. Many thanks to K. O'Regan, S. Chokron, M. Auvray and C. Schoonover for fruitful discussions, and to C. Costello for linguistic support.