Cognitive mechanisms sustaining reading, viewing and working memory are mostly independently examined. In this study, we investigate their interaction in a visual-world priming task. Participants first read relative clauses (RC) morphologically disambiguated for high-(to NP1, HA) or low-attachment (to NP2, LA) (e.g., The helper|helpers of the bakers|baker who will [sg] deliver the bread has|have arrived), and then heard a spoken temporarily ambiguous RC (e.g., the father of the baby who will drink the beer|baby bottle is tall) while presented with a visual context (i.e., VWP). Using linear-mixed effects models, we predict anticipatory fixations to the visual referents associated with NP1 and NP2 as a function of: (i) second-pass time observed during their reading and (ii) individual working memory scores. We demonstrate that high-capacity individuals anticipate more the (non-primed) visual referent when they reread more its associated NP (e.g., anticipate the visual referent 'father' when reread more often NP1, the helpers). We suggest that working memory capacity allows individuals to maintain alternative syntactic analyses of the sentence, and evaluate them upon a subsequent visual context. These findings provide support to the constraint-capacity theory, and shed new light on the cross-modal mechanisms underlying syntactic ambiguity resolution.
working
As we read, words are attended and integrated into
sentences
To preview our work: we utilize data from a structural priming task (detailed in section Methods), where participants were screened for verbal (reading and backward-digit spans), and non-verbal (spatial span) WM tasks. Then, in an eye-tracking experiment, they first read temporary ambiguous relative clause sentences (RC), disambiguated for high-(HA) or low attachment (LA), and were subsequently presented with another spoken RC, but this time concurrently with a visual context. Building on the capacity theory, we hypothesized that high WM individuals can maintain in memory multiple analyses of the sentences read and, therefore, evaluate them on the visual context once attachment ambiguity needs to be resolved during situated language understanding. To test our hypothesis, we examine anticipatory looks to visual objects corresponding to each of the primed attachments (HA, LA), as a function of associated reading time to NP1 and NP2, and to the scores of working memory tests. We find high-WM individuals to anticipate more the visual object associated to the alternative RC (non-primed) interpretation, when they reread for longer the corresponding NP. These results give support to the capacity constraint-theory, and go beyond it, by uncovering the shared cognitive mechanisms bridging reading, viewing and working memory.
Before going into the details of our study, however, we contextualize our research hypothesis within the literature on reading and situated understanding of structurally ambiguous sentences.
Structural ambiguity has always played a pivotal role for
research on sentence processing and theories of syntactic
parsing.
Important to the present work is the study by
Evidence for attachment preferences of RCs on
eyemovement can also be found in situated language
understanding.
Eye-movements are a sensible index of the interpretative strategies adopted as ambiguous RC are either read or understood in a visual context. Moreover, WM can be a key indicator of readers’ capacity to retain and evaluate multiple analyses of attachment resolution, as well as an important component for situated language understanding. To the best of our knowledge, attachment resolution of RCs has been studied independently in reading and during situated 1 But see Swets, Desmet, Hambrick & Ferreira (2007) for inconsistent evidence on WM’s mediation of ambiguous RCs’ attachment in an offline task. language processing. Reading of RCs may be informative of attentional patterns occurring when these structures are understood, by the same individual, in a visual context. In addition, WM capacity can be involved on acquisition of structural information during reading and on its pro-active re-use during situated understanding.
We investigate how reading times on temporary ambiguous RCs (HA and LA) can predict anticipatory eye-movement during spoken understanding of the same structure situated in a visual context. Most importantly, we examine how WM capacity mediates the transfer of structural information from reading to viewing in a visual-world paradigm (VWP) task.
Portuguese speakers have a preference for low-attachment
of RCs
We utilize a subset of data collected in a previous study
which investigates syntactic priming of RCs
High attachment: to NP1 (Low attachment: to NP2) O irmão (irmãos) dos herdeiros (do herdeiro) que vai ler o testamento é (são) de França The brother (brothers) of the heirs (heir) who will [sg] read the will is (are) from France High attachment: to NP1 (Low attachment: to NP2) O ajudante (ajudantes) dos padeiros (do padeiro) que vai distribuir o pão chegou (chegaram) The helper (helpers) of the bakers (baker) who will [sg] deliver the bread has (have) arrived
High attachment: to NP1 (Low attachment: to NP2) O pai do bebé que vai beber a cerveja (o biberão) é alto
The father of the baby who will [sg] drink the beer (the baby bottle) is tall (ditransitive verbs and conjoined constructions), were fillers, interleaved with the experimental items. From this dataset, we only consider the subset of data where two sentences with the same structure3 were read (i.e., 2 primes condition). The same participants were also screened for verbal (reading and backward-digit spans) and non-verbal (spatial span) working memory capacity prior/after (counterbalanced between-participants) the eye-tracking session.
A pre-test on our target sentences corroborated the Portuguese preference for low-attachment. Participants (N=15) read 96 sentences (48 sentences in both HA and LA versions, presented in two random orders), and were asked to tell who was the agent of the action (i.e., the father or the baby in (2)), and score the acceptability of the sentence (on a Likert scale form 1 (totally unacceptable) to 7 (totally acceptable). A low-attachment preference was substantiated by higher accuracy in the LA condition (mean 0.95, sd = 0.21) compared to the HA condition (mean 0.58, sd = 0.49). Sentence acceptability, however, did not depend on its attachment type: ratings for LA (mean 5.1, sd = 1.69) were not significantly higher than HA (mean 4.4, sd = 1.78), as indicated by a Wilcoxon Signed-Ranks Test (Z = -1.24, p = 0.22). Additional evidence for this preference is found also in reading. In particular, an analysis of regressions into NP1 and NP2 as a function of AOI (NP1 vs. NP2) and sentence type (HA vs. LA), shows main effects of sentence type and AOI, whereby readers regressed more often in HA sentences and into NP1 (β = -0.09 and β = -0.33, both p < 0.05).
Participants’ eye-movements were recorded using an SMI IVIEW X™ HI-SPEED eye-tracker at a sampling rate of 1250 Hz on a 21” screen (1024 x 768 px. image resolution). Viewing was binocular but only the participant’s dominant eye was tracked (determined by a prior parallax test). Connected to the participant PC was a satellite speaker and subwoofer system for auditory presentation. A 5-point calibration was done before the experiment began and was repeated every 4 critical trials (20 sentences) or whenever the experimenter found it necessary (half of these included a validation process where we accepted the calibration for angle deviations smaller than 0.5/1 for x and y respectively).
Participants’ working memory was assessed through verbal
(reading and backward-digit) and non-verbal (spatial) WM
span tasks
In the reading task, the sentences were presented for 5 seconds. After each sentence, participants had to make a grammaticality judgment and, after the end of the set, they had to recall the final word of each sentence read, in the correct order of presentation4. In the backward-digit task, the sets consisted of digits from 1 to 9 (in sequences drawn from a table of random numbers), presented one at a time for 700ms. At the end of each set, digits should be recalled in reverse order of presentation. In the spatial span task, participants were presented with sets of letters, one at a time, for 3 seconds. Each letter could be normal or mirrorimaged, and appear in different orientations (i.e., rotated in 7 possible angles). Participants had to decide, after each letter, if it was normal or mirror imaged; and, after each set, recall the original orientation of each letter of the set in the correct order of presentation.
In all tasks, a recall prompt signaled that the set had ended
and that the participant should recall the items. The duration
of this prompt increased as a function of the set size. For
each task, the total number of correctly recalled items
(ranging from 0 to 70) by one participant was taken as that
participant’s score on the task. For each participant, we
computed a composite measure based on the average of the
scores of the three tasks, and standardized it into z-scores.
This procedure increases reliability of the scores, and their
generalizability across tasks
Ambiguity resolution is differently characterized by eyemovement in reading and during situated understanding. During reading, eye-movement responses index processes of attachment interpretation and re-analysis (e.g., re-reading previous ambiguous regions5). During situated understanding, instead, they index anticipatory mechanisms evaluating the object in the context which most likely resolves the attachment (e.g., fixation to S1 for an HA interpretation upon hearing who).
From the reading data, as AOIs, we consider the two noun
phrases referred by the pronoun, NP1 (The helper|The
helpers) and NP2 (of the bakers|of the baker). From the
VWP data, as AOI, we consider the corresponding depicted
referents, S1 (The father) and S2 (the baby), refer to Table 1
for example material. We focus on these linguistic and
visual AOIs to establish clear links between the referents
read, and their counterparts in the visual context.
Reading Measures: We consider second-pass on NP1 and
NP2, as the sum of all fixations duration, beginning with the
first reentry in the region and ending when the reader leaves
it, in any direction. Length-residualized values, i.e., the
differences between predicted (by linear regression as
function of word length) and observed values, were
computed for each subject, to account for word variability
Visual-World Measures Six visual objects were defined in the array by drawing annotated polygons labelled as: S1 and S2 (the antecedents of the pronoun for HA and LA, respectively), O1 and O2 (the objects acted upon by the subjects), Central (accounting for center bias) and Distractor (an unrelated object). Fixation coordinates from the tracker output were mapped onto these areas.
For each trial, fixations were aligned, item-by-item, to the onset of the pronoun who, creating a time window up to 400ms after it7 (with 200ms added to account for oculomotor programming). Fixation points were aggregated in 50ms bins, and the percentage of fixations on each visual object was calculated relative to the total amount of fixation on every other object. We took the mean percentage of fixations across 8 bins (i.e., 400ms), and obtain a unique measure of fixation to correlate with the second-pass observed in the same trial (refer to Figure 1 for the trial run).
Our dependent variable is percentage of fixations on S1
(and S2), predicted as a function of reading NP1 (and NP2),
prime type (HA vs. LA) and WM z-scores in a
maximalrandom mixed effect models
In Figures 2 and 3, we scatter percentage of fixations to S1 and S2, as a function of second-pass readings on NP1 and NP2 respectively, for high and low WM groups8 in the two priming conditions (HA - left panels, LA - right panels). We observe that fixations to S1 and S2 are differently modulated by second-pass reading times on NP1 and NP2, the type of Prime, and WM. In particular, starting with fixation to S1, we observe a significant two-way interaction between prime and second-pass, whereby stronger anticipation to S1 is observed with longer second-passes, especially when the prime is LA (two-way interaction prime:secondpass, refer to Table 2 for the model coefficients). Moreover, this effect is especially strong in individuals with high WM (prime:secondpass:WM, visualized in Figure 2, right LA panel). This result shows that memory capacity enables comprehenders to encode and recall the HA alternative when primed with the preferred 7 See Arnold, Eisenband, Brown-Schmidt & Trueswell (2000) for evidence of rapid anticipatory eye-movement as indexes of pronoun resolution in a visual-world task.
8 Note the WM grouping (considering two halves of the distribution) is only for purposes of visualization; the WM scores were introduced as a continuous measure in the model. LA structure. Most importantly, this happens when they reread for longer NP1, indicating that they evaluated an HA interpretation of the sentence. Fixations to S2 corroborate this result. Here, we observe less anticipatory looks to S2, for longer second-pass reading NP2 in LA primes, for highWM individuals. Our results can be motivated by the capacity-theory of sentence processing, where WM holds multiple possible representations for the same sentence.
The interaction, however, was stronger on fixations to S1 with respect to the second-pass readings on NP1. This result might reflect the preference of Portuguese readers for LA. In fact, percentage of fixations to S2 (LA interpretation, Figure 3) is overall higher than anticipation of S1 (HA interpretation, Figure 2), reflecting a bias towards LA. In Just & Carpenter’s theory, the higher WM is, the more the individuals can construct multiple representations when faced with an ambiguity, which levels of activation are determined by other factors such as the frequency of occurrence in the language. In our data, high-WM readers, as expected, show less difficulty with the dispreferred high(Intercept) prime secondpass (NP2) WM prime:WM prime:secondpass secondpass:WM prime:secondpass:WM
Percentage of Fixation on S2 (400ms after who)
Estimate SE t
42.400 3.570 11.878
-1.774 3.811 -0.465
-0.006 0.011 -0.524
-0.336 3.529 -0.095
-1.285 3.962 -0.324
-0.034 0.021 -1.617
-0.013 0.011 -1.133
-0.042 0.021 -1.961
attachment; they do not have to re-inspect NP1 in reading
HA to further anticipate S1 (additional analysis on residual
second-pass on NP1 as a function of WM and Prime
revealed a main effect of WM (β = -24.61, p = 0.036),
confirming that HA sentences are not so demanding for
high-capacity readers). Yet, the capacity model assumes that
the less frequent alternative (here HA) should have an a
priori lower level of activation, even when high-memory
allows for its activation. Why, then, should high-WM
readers strongly anticipate S1, rather than S2, when
presented with a LA sentence? We suggest that memory
capacity not only allows for encoding of multiple
alternatives but also leads to strategic strengthening of less
frequent representation, to avoid future prediction errors, as
proposed by learning accounts of priming
Important advances in theories of sentence processing
have been made by investigating visual attention during
reading
In the present study we provide, for the first time, linking evidences between online reading measures of sentences and online measures of their incremental processing when situated in a visual context9. Furthermore, we aimed to reconcile the dynamics of these processes with working memory capacity. We did it by showing that ambiguity resolution strategies at reading can directly inform on anticipatory mechanisms during language understanding, and are mediated by WM capacity.
Our results lend support to parallel processing accounts of
syntactic ambiguity resolution and, in particular, to
Fundação para a Ciência e Tecnologia under awards number SFRH/BD/72307/2010 to EF and SFRH/BDP/88374/2012 to MIC, is gratefully acknowledged.