It is well-established that spatial attention can be allocated as a gradient that diminishes from a central focus. In this paper we consider auditory attention and we develop a model for how it is distributed in space following basic ideas of top-down and bottom-up attentional control from verbal models [6, 12]. There are three main components of our model: a goal map, a saliency map, and a priority map. The goal map models the distribution of attention which is allocated by choice (top-down component). The saliency map, as the name suggests, models attention related to the saliency of auditory stimuli (bottom-up component) and the priority map synthesizes the other two maps in an overall distribution of the attentional bias. We model the three maps and their interaction using the well established AI framework of constraint satisfaction problems. We study several hypotheses on the maps and we contrast the results in terms of data obtained running different kinds of experiments. Our computational model, is to the best of our knowledge is the first which targets specifically the auditory system. Our constraint-based approach is very flexible in terms of embedding and testing different hypotheses on the components and constraint propagation techniques allow both to focus on single components as well as to consider the system dynamics as whole. The predictions arising from our model well fit the experimental data, are cognitive plausible and provide new interesting insights to the mechanism of attention control.
Audition is distinguished from the other senses by the ability to panoramically monitor the environment for things happening at a distance, behind obstructions, and out of sight. These considerations make the auditory system particularly useful for shifting attention to events that are important to survival and reproduction. For example, hearing the sound of a snapped twig from a predator hiding in the brush can quickly elicit a fight or flight response. The overall goal of this project is to better understand at the cognitive and neural levels of analysis how auditory attention is allocated over space. The focus is on the interplay between top-down and bottom-up spatial attention biases that govern shifting attention to distractors during performance of a simple spatial attention task. We take an interdisciplinary approach by using behavioral and neural stimulation methods to test and refine a computational model of auditory spatial attention.
Previous work has established the idea of an auditory attention gradient [
Our recent work has defined behavioral and neural measures of auditory attention
gradients [
Our aim is to develop a rigorous theory of auditory spatial attention that relates to current work on dorsal and ventral neural attention systems. We foresee that our work will help advance the understanding of basic issues in attention regardless of modality, such as top-down and bottom up interactions, capacity limitations, vigilance, and will inform debate on supramodal attention processing in the brain. There are also multiple applications, such as topics in human factors, improved audio communication systems, and brain-computer interface control using spatial attention.
Our lab studies aspects of hearing that are particularly important to humans, such
sound location, speech, and music, and how auditory processing is affected by
attention, short-term memory, and action planning and execution. The common
denominator is that these studies contribute to an emerging framework termed cognitive hearing
science, which examines the role of the auditory system in higher-level cognition and
action [
We combine these novel parametric behavioral measures to map-out auditory
attention over space with a computational model to explain how specific top-down and
bottom-up mechanisms jointly determine the shape of auditory spatial attention
gradients. Recent modeling work focuses on saliency, particularly when there is more than
one sound happening at the same time [
In this sections we provide a brief background on psychology literature related to spatial attention and its computational models. We also give some fundamental information concerning constraints. 2.1
Spatial Attention
Almost all attention models from the inception of Psychology as a formal science have
distinguished attention that is directed by personal choice from attention that is directed
to an event by virtue of it having a salient property, such as a loud sound [
Attention can be expressed as a spatial gradient relative to an attended location
[
Auditory spatial cuing decreases reaction times to subsequent targets at a cued
location relative to uncued locations [
Constraints and Computational models of auditory attention
Computational models of cognitive processes are beneficial because they require an
explicit theory, can reveal hidden assumptions or logical inconsistencies, and
simulations can establish proof-of-principle much faster than pilot experiments [
Constraint programming [
Solutions are found by searching the solution space either systematically, as with backtracking or branch and bound algorithms, or use forms of local search which may be incomplete, that is there is no guarantee they will return a solution. Systematic methods often interleave search and inference, where inference consists of propagating the information contained in one constraint to other constraints via shared variables.
The rich variety of finely-tuned algorithms available for constraint problems has made the effort of translating real world problems into this framework an efficient solving approach.
Constraints have been used before in the context of human cognition for example
to model skilled behavior [
Figure 1 depicts the overall hypothesis on the interplay between top-down and bottomup spatial attention processing. There are three main components: goal map, saliency map, and priority map. Each map is a 1-D vector of attentional bias in normalized units (0-1) across the semicircular horizontal frontal plane (from -90 on the far left to +90 on the far right, 2 increments, as shown in Figure 2). The given inputs to the model are (1) attended location, which is a goal map parameter and (2) sound location, which is input to the saliency map. The output is a priority map representation of attentional bias across the 180 semi-circle (in 2 increments). Areas of greater attentional bias are assumed to relate to measurable data by having faster reaction times, more sensitive sensory thresholds, and increased accuracy relative to locations with less bias. We emphasize that this is a model of information processing at the cognitive level. It is designed to help interpret behavioral results and inspire new experiments to test and refine the model. It is not intended to model how neural activity relates to attention. The gray boxes show inputs and outputs that interface with other cognitive functions.
Our computational model adopts a constraint-based approach to cast the interactions
among the three maps into a constraint solving problem that can be efficiently solved
with the rich algorithmic machinery which has been developed for constraints[
We remark how this constraint-based representation is very flexible in terms of modeling different hypothesis on the attentional bias distributions and on the interaction of the maps. In our initial setting, for example, we don’t have any interaction between the goal and the saliency map. However, we could model it in a straight forward way simply by defining new constraints that connect variables of the two maps. Moreover, this
+90 highly decoupled model, which has one variable per location, allows for the implementation of local phenomena, as, for example, we foresee may occur in case of habituation to stimuli coming from a fixed location. More broadly, the ability to model various types of interactions among these subprocesses is relevant to issue of modularity in human cognitive systems.
In what follows we denote with VGi , VSi and VPi respectively, the i-th variable of the goal, saliency and priority map. We note that i ranges in f-90,...,+90g, and the domain to quantify attentional bias uses normalized units (0-1, in .01 increments). The goal map indexes top-down attention bias, and is a function of the central executive in verbal models. It models top-down, voluntary focus of attention to a location, and has a progressive, symmetrical decrease in attentional bias away from the attended location. We currently consider three options for modeling the attentional bias in the goal map given that location A = a is (voluntarily) attended. We express them as a sets of constraints each of which is defined over variable A and VGi . In what follows we indicate the tuple of values which is allowed by the constraint.
– Standard Gaussian Distribution: (A = a; VGi = GGe 2jad2i 2 j G ) where dG is the standard deviation of the goal map and G is the height of its peak. – Modified Gaussian distribution with inhibition: (A = a; VGi = (GGe 2jad2Gi12 )) + (GG j
GGe 2jad2Gi22 )); j notice that this obtained as the sum of a Gaussian and an inverted Gaussian. G is the maximum of the two functions and is a parameter that we use to weight the A"en%on Loca%on
Constraints
Goal Map Variables
Saliency Map Variables
Priority Map Variables components. We also have two standard deviations for the components which are denoted by dG1 and dG2. In this way we obtain the desired shape, which has a peak at the attended location, then dips down to an area of lower attentional bias and then increases and stabilizes as we move far away from the attended location (see Figure 4 (a)). – Constant function:
(A = a; VGi = k); where k is a constant value.
The different shapes for the goal map when the attended location is 0 are shown in Figure 4 (a). We note that in Figure 3 the standard Gaussian distribution is shown on top of the variables corresponding to the goal map.
Similarly we consider two options for the saliency map, which models how attention is allocated to a stimulus given how salient its characteristics are. Again, in our model, this amounts to defining constraints between variable A and each saliency map variable. The two options are: – Inverted Gaussian distribution: (A = a; VSi = GS
GS e 2jad2ij2
S ) where dS is the standard deviation for the saliency map, and G is its maximum value. – Constant function:
(A = a; VSi = k); Inverted Gaussian
Constant 0° (a) -90°
and between 0 and 1.
We elaborate on the cognitive interpretation of these hypothesis in Section 5. 4
We developed a behavioral task to map attentional gradients and to test the above model.
It is a hybrid of our spatial target detection task [
Figure 5 plots reaction times x location for each standard condition in absolute space (A), as well as the deviant location relative to the standard location (B). There were two main results. First, all conditions had slower responses to distractors vs. standards (p < :001), indicating attention shift costs. The reaction time x location function is more prominent for the left vs. the right standard (p < :01), suggesting that it is faster to shift auditory attention from right-to-left than from left-to-right. The 0 standard has an increase at near 45 locations, similar to the left standard, but a decrease for the 90
Fig. 5. Basic Attention Task: Reaction Time Results & Modeling locations, similar to the right standard (p < :001). Accuracy was very high (> 95%). The basic results were replicated in new subjects (n = 12; p < :01). (C). Second, in each condition reaction times sped-up for the farthest distractor location (p < :001). This was seen in each subject’s first block, so is not due to carry-over effects from previous standard locations. The faster responses at far distractors cannot be accounted for by a graded reduction in bias from the attended location (goal map alone). Instead, the heightened bias to far distractors is modeled by the saliency map. A control condition in new subjects and equal probability at all locations had no differences in reaction time (n = 20; p = :83), ruling out accounts based on perceptual differences among locations. In D the inverse of the reaction times is given to show how attention bias is theorized to relate to reaction time (greater bias ! faster reaction times). 5
We have considered two combinations of options for the goal and saliency map and we have compared how well the emerging priority map fits the behavioral experiments data. The combinations that we have considered are – Hypothesis 1: standard Gaussian centered for the goal map and inverted Gaussian for the saliency map, both centered at the attention location; – Hypothesis 2: modified Gaussian with inhibition for the goal map and inverted Gaussian for the saliency map, both centered at the attention location;.
We recall that the goal map models the voluntary focus of attention. Both options that we consider model a peak of attention around the attended location and then a symmetrical region of lower attentional bias away from the attended location. In addition, the modified Gaussian assumes an area of inhibited attention around the peak. As far as the saliency map, the inverted Gaussian shape is consistent with results which we have observed in our experiments above. which suggest that bottom-up attentional bias progressively increases away from the attended location.
We have, in addition, assumed the same maximum level G = GG = GS for the goal and the saliency map. This corresponds to saying that peak attention levels generated by the top-down and bottom-up component are the same, which is reasonable, in particular if the components are thought of as independent of each other. Another similar constraint which we plan to consider in the future is fixing the overall amount of attentional bias to be constant across the maps.
We also assume = , thus taking the view that the top-down and bottom components equally contribute to the overall attentional bias. We call this parameter . We have fitted the data by using a stochastic local search approach on the parameters of the functions, which we recall are: G, dG, dS and for hypothesis 1 and G, dG1, dG2,dS , and for hypothesis 2.
The evaluation function we have used is the sum of squared errors:
E(p) =
X where dx is the bias associated to location x in the experimental data and p(x) is the value associated to x by the priority map. We performed the fit for all three attended locations, that is, -90 , 0 and +90 .
The best results for all three locations have been obtained by Hypothesis 1 with fitting values (1 E(p)) equal to: 0.943 for 0 , 0.739 for +90 and 0.904 for -90 . The corresponding fitting values for Hypothesis 2 are instead: 0.903 for 0 , 0.501 for +90 and 0.850 for -90 . As it can be seen Hypothesis 1 outperforms Hypothesis 2 in particular in the +90 case. This is in part due to the asymmetry between +90 and 90 data. However, Hypothesis 1 fits better in both cases despite being symmetric. In Figure 6 we show all the maps corresponding to the best fit superposed with the data. 6
We have a rich agenda of future directions.
Impact on goal map of short-term memory load. We hypothesize that, relative to no load, the addition of a short-term memory load, such as first memorizing three words and then performing several trials of the attention task, increases reaction times to distractors near the attended location, decreases reaction times at far locations, and increases inter-trial variability. Future experiments will assess the role of short-term memory loads on the goal map. We conjecture that memory load should impair topdown control as task-specific information and load both rely on short-term memory. The rationale is that the predicted reaction time effects would be in range of the goal map but not at far locations (relative to the attended location) which are mediated by the saliency map. Load effects will be modeled by introducing a probability distribution over the goal map options and assuming that memory load results in some trials with equal attentional biases across locations (constant function). This should also increase reaction time variability. Loudness and attention gradient. We plan to investigate whether intensity changes decrease reaction times near the standard location, with progressively smaller increases at more distant locations. Loud sounds induce automatic orienting, which our model would represent with the saliency map attentional bias. Saliency, and attentional orienting, can also follow from the absence of an expected sound, such as an engine unexpectedly stopping (or give a better example). The study will test whether attentional bias due to changes in stimulus intensity follows from saliency (increases or decreases bias attention) or loudness (only increases bias attention). Behavioral experiments will distinguish between an alternative hypothesis that intensity changes slow responding due to shifting attention. We expect locations near the standard to receive little benefit from saliency due to proximity to the goal map focus, and would be enhanced by intensity-based saliency bias. At far locations the saliency map is already tuned to bottom-up inputs, and has less to gain by intensity changes. Intensity effects are expected to vary by standard location ( 90 > 0 > +90 ). The basic task will be used with manipulations of stimulus intensity. Intensity effects will be modeled by the introduction of new variable having as values the intensity levels of the stimulus. We will modify the inverted Gaussian option for the saliency map with a new parameter obtainja ij2 ing K(G Ge 2 d2S ), where the new parameter is denoted by K. Moreover, we will add new constraints connecting the location variable A, the new intensity variable and the saliency map variables.
Location probability and attention gradients. Finally we plan to understand if, as we
conjecture, the probability of a stimulus at a given location is negatively associated
with reaction time. It has been shown that attentional bias is strongly dependent on
expectations (Itti and Baldi, 2009) and that the degree of expectation depends on base
rate, with unlikely events having large evoked potentials [
We have presented a constraint-based model of auditory spatial attention. Our model is based on a well established decomposition in top-down and bottom-up components and, to the best of our knowledge, it is the first one focusing on the auditory system. Constraints allow for a high degree of flexibility in terms of hypotheses testing. Our initial results in terms of fitting experimental data are very promising and bring interesting insight on the role and interplay of the two components in the distribution of auditory attention in space. Acknowledgements. This work is supported by NIH under grant number R01-DC015736.