We present an overview of speech research applications of gaze tracking technology, where gaze behaviours are exploited as a tool for analysis rather than as a primary object of study. The methods presented are all in their infancy, but can greatly assist the analysis of digital audio and video as well as unlock the relationship between writing and other encodings on the one hand, and natural language, such as speech, on the other. We discuss three directions in this type of gaze tracking application: modelling of text that is read aloud, evaluation and annotation with naïve informants, and evaluation and annotation with expert annotators. In each of these areas, we use gaze tracking information to gauge the behaviour of people when working with speech and conversation, rather than when reading text aloud or partaking in conversations, in order to learn something about how the speech may be analysed from a human perspective.
Gaze tracking is used in a number of applications that aim to improve interaction
between humans or between humans and machines. Examples include interfaces that
utilize gaze tracking to allow hands-free pointing, to track the addressee of speech, or that
utilize pupil dilation to track cognitive load, for example in drivers. There is a wide
range of such application areas and more. Gaze has been associated with for example
cognitive state, cognitive load, direction of visual attention and turn taking in
conversation
More recently, a number of new applications of gaze tracking have surfaced in which gaze tracking is used to model the behaviour of people who are somehow working with or analysing actions (e.g. speech), rather than people who are in the process of performing them (e.g. partaking in a conversation).
In the following, we describe three main areas where gaze tracking is exploited in this relatively new manner, as a tool to assist in the analysis of speech and language, rather than as an object of research in itself: Modelling of text to be read aloud Evaluation and annotation with naïve informants Evaluation and annotation with expert annotators We discuss the potential benefits and the risks involved, and highlight the particular requirements placed on gaze tracking technology by these specific application areas, as compared to other areas where gaze is used, such as in experimental psychology and in interaction design. 2
Reading texts aloud is an important application area of speech technology. Apart from its use in commercial applications, speech synthesis, or text-to-speech (TTS), is used by government authorities to produce talking books as a means of making texts accessible to those who are for some reason not able to read in the traditional sense of the word. As an example, the Swedish Agency for Accessible Media has produced thousands of talking university text books and continuously produces around a hundred talking newspapers using TTS. The impact of less-than-perfect TTS, then, is great, but we still struggle with finding viable ways of modelling how text should best be read aloud.
A key characteristic of speech is that it exhibits variation in prominence. Speakers
make some words more prominent by lengthening and by expansion of spectral
characteristics, whereas other words are more backgrounded, that is, reduced. A number of
models have been suggested to help assess which words in a text that is to be read aloud
should be made prominent, including measures based on the probabilistic aspects of the
text
Behavioural research has long been using gaze data to study reading, which has illuminated salient linguistic cues used by skilled readers. Word frequency, neighbourhood frequency, and syllable length have all been associated with our reading behaviour and with our processing of textual information. It is therefore likely that gaze data can refine, compliment or substitute statistical models of text.
A key difference between gaze behaviour and language models is that the latter,
while efficient in capturing predictability from a within-text point of view, they do not
capture the processing steps as they occur online while a person is reading. Gaze data
has also been shown to correlate with subjective measures of word predictability in
sentences
In ongoing studies, we are attempting to model prominence via gaze, by collating
gaze tracking data from multiple readers of the same text to determine to which words
readers pay particular attention. We hypothesize that quantified gaze data can
accompany and strengthen models based on sample distribution of contextual predictability
of words in the text – statistics which frequently form the basis for prosodic models.
Others have also pointed out that future prosodic modelling with gaze or other similarly
inspired applications could emerge as alternative methods for modelling prosody in
TTS
Speech science and speech technology research are increasingly data driven, and today, the greatest bottleneck for machine learning of speech and human interaction behaviours is no longer limited access to speech data, of which there is a lot, but rather access to useful annotations of speech data. Acquiring good annotations, or labels, on which to train models is expensive and time consuming, and developing efficient methods for these purposes is becoming increasingly important.
In addition, speech technology research is in most cases an iterative process, where a method is developed and trained, then evaluated, and then retrained using more or better data. The results of many evaluation results can be viewed as a series of examples of what went well and what did not work. Seen from another perspective, this is training data, and it is often the case that the results of an evaluation that ends an iteration become the input for the training of the next iteration. Evaluation and annotation, then, are closely related.
Gaze tracking has been used both for evaluation and annotation in speech science. Here, we discuss two methods that operate on naïve informants in the sense that the informants are not professionals, nor have they received any special training to complete their tasks. Instead, these methods tap into more or less subconscious human communicative behaviours. The first method evaluates TTS quality by measuring gaze patterns of informants responding to instructions during an audio-visual task, and the second tracks the gaze target of people observing recorded interactions to learn more about speaker changes in conversation.
Most TTS evaluation methods fall short when it comes to pinpoint where in the evaluated speech a problem occurs. When informants fill in a questionnaire after listening to TTS, they are not able to point to particular instances of the speech to which they listened, but rather make general judgements. Gaze data, on the other hand, is synchronous, meaning that it may give insights about a person’s perception at the same moment it takes place.
Swift, Ca
Van Hooijdonk, Commandeur,
More recently, White, Rajku
When it comes to speaker changes, a recurring problem with data from real conversations is that one cannot be sure that a place where a speaker change occurred was actually a suitable place just because it occurred, as people sometimes flaunt conventions, and similarly one cannot be certain that a place where no speaker change occurred was an inappropriate place for a speaker change.
Based on the observations that lookers-on of a conversation fixate the speaking
person and redirect their gaze in expectation of a speaker change
Trained professionals are paid to annotate data. The standard way of getting transcriptions of speech, for example, is to pay transcribers to write down what is said, usually painstakingly following a detailed transcription manual. Other tasks are performed similarly. Phonetic segmentation, for example, is the task of splitting utterances into their phonetic units. Phonetically segmented speech is used for a wide range of purposes in speech technology development, including the training of ASR and TTS. Automatic segmentation – or forced alignment – does this job very well for some recordings, but in other cases manual labour is still required to reach acceptable quality.
Manual phonetic segmentation is an example where data sets on expert annotator
gaze behaviour can add new layers of information to training data for developing
automatic methods. Khan, Steiner, Sug
In most fields where gaze tracking has been employed, there is a need to pay great attention to aspects such as control, exactness (which requires precise calibration), and clear instructions. Often, the goal of the tracking makes it necessary to know, for each moment in the experiment, exactly where the gaze rests. Conversely, the applications we discuss here generally require large amounts of statistical data, where ill effects of a small proportion of errors may be less damaging. On the other hand, ease of use is important (or the professionals will not agree), and calibration must be unobtrusive or even hidden (lest the presence of the instrument endangers the ecological validity of the experiment). In general, the requirements of these applications are more concerned with usability issues and less with experimental control. 6
In summary, we believe that tracked gaze behaviours can boost a wide range of speech technology and spoken interaction research methods. However, gaze applications of this nature place different requirements on the gaze tracking technology, and the methods are still in their infancy.
Our work for the near future focusses on modelling of text to be read aloud with TTS and on further exploring the possibilities afforded by registering 3rd party observers’ gaze behaviours. We predict, however, that as gaze tracking hardware becomes increasingly accessible, the range of applications of the type discussed here will quickly grow.