erst¨ Univ ogt -V h sytem blind the asitng inspred y b td presn tesd A and b impared with visualy effns ectiv to facilte haptic wledg kno asitng tacile devic. trled con discu expts rimn a a [3 sytem c , tha genrats erp otd y prote user uterancs en giv tacile eralizd as virtual tacile effns ectiv [1 tha as w explors maps alutd ev of h suc 0 lop dev acording actile T e w subtie maps
, M a t t h i a s K e r z e l a n d C h r i s t o p h e r H a b e l n t n t { l no-visual increas ey
wo
rd M s aps, ulti-M
o dal erbaly V M aps,
n Maps are imp ortant external urban-area maps in every-d bakery or physician in enviro limited for blind and visually substitutes. Acquiring knowl by vision: On the one hand, i than that provided by visua b e integrated over time. To maps, providing additional 11] we suggested that appr by developing a system im Tactile Maps (VAVETaM). assisting utterances, which the prop er names of the m drawbacks stemming from utterances are based on a representations of space. F ay scenarios such as plann nments novel to them. The impaired p eople, therefore, edge from maps haptically nformation provided by tac l maps; on the other hand reduce some difficulties in information via the auditor oaches towards audio-tactil plementing Verbal ly Assisti The suggested system gener not only inform the explorer ap ob jects, but include more the sequentiality of tactile-ma corpus study (see [9] for a or example, humans use ing a route to the next access to visual maps is tactile maps are used as is different from doing it tile maps is more sparsely , this information has to haptic comprehension of y channel is useful. In [3, e maps can b e extended ng Virtual-Environment ates situation-dep endent of the tactile map ab out information that reduces p reading. The generated discussion of the corpus P r o c e e d 2 and the set of assisting utterances). For example, the assisting utterances include information about spatial relations between map objects. The Sensable Phantom Omni device allows the map user to perceive virtual tactile maps (see Fig. 1). The device can be thought of as a reverse robotic arm that generates force depending on the position of the pen-like handle. Streets and landmarks such as buildings are marked as indentations that can be felt with the device.
In this paper, we present a prototype implementation of the VAVETaM system. The goal of the development of this prototype is to show the technical feasibility of the suggested system; that is, that the prototype works. The feedback gained from the user study presented in this paper will be used for further development.
In the remainder, we discuss relevant literature (Section 2) and we briefly introduce the prototype (Section 3). Furthermore, we present a user study with the prototype (Section 4) before concluding the paper. 2
State of the Art
Audio-tactile maps as improvements of uni-modal tactile maps have successfully
been developed and tested for usability. Some approaches use physical (‘printout’)
overlay maps on touch screens (e.g., [13, 14]). The Phantom device used in our
prototype has been successfully used to present virtual tactile maps with
audiofeedback [
We have positively evaluated the multi-modal interface that is described here prior to implementation in Wizard-of-Oz-like experiments (i.e., the assisting utterances were controlled by the experimenter) with both blindfolded sighted and visually impaired people [10, 7]. Participants received assisting utterances as suggested compared to a condition in which they were only informed about the names of objects. Spatial knowledge acquisition of virtual tactile maps was facilitated by verbal assistance. 3
The Prototype System
To generate assisting utterances, two important tasks have to be solved. First, the stream of position information from the haptic device is analyzed in order to detect semantic—that is, meaningful—exploration events. Haptic event detection, for example detects when a map user explores a street with the haptic device. This information is the input to the generation component. Second, based on this input, appropriate information is selected from a knowledge base and prepared for verbalization; that is, natural language is generated.
For each of these tasks, components were developed. Haptic event detection is described in more detail in [6, 5]. A generation component and the interface between event detection and generation is discussed in [9, 8].
We connected the two components to a fully working prototype system. The utterances produced by the prototype are similar to those evaluated in previous experiments (see Section 2). For example, when the movements the user performs with the device indicate that he or she is interested in a street called ‘Amselweg’, the system produces an output which can be translated as follows: ‘This is Amselweg. This street is parallel to Blumenstraße. It forms a corner with Dorfstraße at the top end and towards the bottom it is restricted by the map frame. Furthermore, it crosses Hochstraße.’ See [9], for a thorough discussion of the content of the utterances. The generation of the utterances is not based on canned text. Both, the verbalization history and the current map exploration, are taken into account: When information was already verbalized, repetitions leading to unnecessary redundancy are avoided. When the map user starts to explore another map object—for example, a landmark called ‘Rathaus’ (town hall)—the verbalization is stopped after the ongoing sentence is finished and the map user receives currently relevant utterances.
The generation system and the haptic event detection are implemented in Java. We use the Chai 3D toolkit for haptic rendering of the virtual tactile map. The Chai 3D toolkit and the haptic event detection are interfaced using JAVA Native Interface. Speech synthesis was realized using Mary TTS.1 1 tp:h//w.hai3cd.org and tp:h//.marydfki.de .
I refd.
t en w , e r z e l ea p o .70 .10 .50 .80 .70 .50 .90 .80 1 . 7 0 1 . 5 0
S
D r o c e e d i n g s otyp e system, we asked 13 participants (university students, rse credit or monetary; mean age: 24.7 years, SD = 7.2 years, prototyp e and to give us feedback by indicating agreement it and by taking part in semi-structured interviews. aining with the haptic device, participants were blindfolded the VAVETaM prototyp e. After they rep orted that they avior of the prototyp e, they were instructed to learn the ould b e able to sketch it afterwards. This instruction was clear goal for the exploration of the map. Participants s they wanted to explore and memorize the map (overall = 14:16 min, SD = 4:46 min; interaction after instruction: 3:42 min). After they sketched the map, they were asked nt to a set of statements ab out the prototyp e. Selected n in Table 1.2 They were given the list of statements in swers were given on a 1–5 Likert-typ e scale (1 corresp onds y’ and 5 to ‘I do not agree at all’). The list of statements her with the exp erimenter in order to enable immediate ant p oints (discussions were audio-recorded). ses to the statements indicate that participants considered to supp ort tactile maps with natural langauge (B) and b ehavior of the prototyp e comprehensive (J). Furthermore, aps were considered understandable (A, F). The resp onses ossible to lo cate landmarks, although with more difficulty ts (G). A p ossible explanation is that the connected network 2
Due of the orse w from: other neutral. tp :h of y prote the pro duce pa er, a The / ful / list of staemn and w.informatk.uni-burg .hamde mean / resp on can WSV/ b e d retiv AVV Verbally Assisted Virtual-Environment Tactile Maps: A Prototype System of streets was easy to follow, while landmarks were not connected with each other and had to be explored one by one.
Participants considered the timing of assisting utterances appropriate to establish reference to the map object to which the assistances belonged (I). They considered the utterances helpful and they indicated that they knew what was meant (D, E). The ratings whether utterances were understandable (C) are comparably low. This seems somewhat contradictory to the high agreement with assertion D. In the discussion and the interview, participants indicated that problems with understandability were in general limited to proper names.
Currently, once sentences are started they cannot be stopped or changed anymore. From our own experience with the system, we expected that this is potentially confusing. In fact, most participants considered the inability of the prototype to change current utterances confusing (H).
In the interviews conducted subsequently, the points discussed above were elaborated and extended. Participants in general considered the set of assisting utterances appropriate and did neither miss important information, nor did they indicate that superfluous information was given. When asked for suggestions on how to improve the system, participants considered options for customization (changing the haptic presentation of the map, switching certain kinds of assisting utterances on and of) and the quality of the synthesis as potential improvements. Some participants indicated that they would have changed the content and frequency of certain utterances—however, this feedback was not consistent. 5
Conclusion We presented the Verbally Assisting Virtual-Environment Tactile Maps (VAVETaM) prototype that solves the task to generate assisting utterances for tactile map explorations. The prototype is based on the interaction of two components: Haptic event detection and assistance generation. The effectiveness of the system was previously evaluated positively in controlled experiments [10, 7]. We presented a user study with the prototype in order to show that it works. Participants considered the prototype behavior comprehensive and the utterances and haptic presentation understandable. Potential improvements pointed out by the users are (1) the quality of the speech synthesis with respect to proper names, (2) the possibility to customize the behavior of the system, and (3) the implementation of a possibility to change ongoing utterances when the user moves to another part of the map. Aside from these potential improvements, both, with respect to the assisting utterances and to their timing, participants considered the prototype system usable, comprehensive, and helpful.
Acknowledgments. The research reported in this paper has been partially supported by DFG (German Research Foundation) in IRTG 1247 ‘Cross-modal Interaction in Natural and Artificial Cognitive Systems’ (CINACS). We thank Martin Christof Kindsmu¨ller for discussion. We thank the anonymous reviewers for their highly useful commentaries. e e e c
s ., Rena,
3 K In: 0. K In:
F.,
onferc. p. 07 L ) eg, 7–358 M., of AA
(200
. , D e ,istan A . : A aplicton G .
E . : ords :w 7 : ) erbal V asitnce tacile-map aps: ards w to 1 . 2 . 3 . 4 . . . . . . . . . . . rendig ( es ,P ark t men Pro cedings disable ang ,W aces erst¨ Univ
y prote sytem. orkshp W on Sph ec
Eba ,sc hen exploratins. In: iudce, N.,
Hab el, C
I., orkshp W on 0) olas ,Nik G IE .: 2 M
D In: visualy oNtingham , maps of the p eol.
Z. digtal 1th 4 “NO A informat partily ted sigh p eol.
(1988) Li, B., ofr p eol with visual t .imparen In: teraol in ACM 3–50. A :
aumn, the
(2 H Ae sitv hnolgis ec T 01 ab el, C p erction lingusM .
(eds. ) Pro
R.
01 ., K
erbal v asitnce faciltes 2 H ., 2 Springe, wledg kno pres) C H 1 : .
M.
(eds.)
1 ) G C. enratig , rbeal v . ., E K E ostp ul, M K ultimeda . : Haptic ”: to l acquiston, mangeT. , Hen ,a v T. : A , C
C AP E ( and ) A .cesiblt y