In this paper we discuss two types of nominal copular sentences (Canonical and Inverse, Moro 1997) and we demonstrate how the peculiarities of these two configurations are hardly considered by standard NLP tools that are currently publicly available. Here we show that example-based MT tools (e.g. Google Translate) as well as other NLP tools (UDpipe, LinguA, Stanford Parser, and Google Cloud AI API) fail in capturing the critical distinctions between the two structures in the end producing both wrong analyses and, possibly as a consequence of a non-coherent (or missing) structural analysis, incorrect translations in the case of MT tools. To support the proposed analysis, we present also an empirical study showing that native speakers are indeed sensitive to the critical distinctions. This poses a sharp challenge for NLP tools that aim at being cognitively plausible or at least descriptively adequate (Chowdhury & Zamparelli 2018).
The main hypothesis of this paper is that sentence comprehension cannot be achieved independently from a coherent structural analysis. To support this claim, we first present a precise structural analysis that is critical for recovering the relevant dependencies within specific constructions, then we will show that the crucial structural properties captured by the theoretical framework are in fact correctly perceived by native speakers, but not revealed by some widely used Natural Language Processing (NLP) tools. This leads to poor performance in tasks like Machine Translation (MT).
This argument seems to us especially relevant in those structural configurations in which a nonlocal dependency must be established: in parsing, for instance, interpreting correctly a whdependency requires that the dependent (the whphrase) and the dependee (the head selecting the wh- phrase as its argument/modifier) are identified, and the nature of the dependence disambiguated (e.g. argument vs. modifier). In (1) we exemplify the special case of a non-local dependency between a wh- PP and a DP it depends on (a co-indexed underscore signals the possible extraction sites, hence the dependent constituent; the diacritic “*” prefixes, as usual, illegal sites): (1) [Di quale segnale]i [i telescopi *_ i] hanno Of which signal the telescopes have scoperto *_i [un’interferenza _ i]? discovered an interference? ‘[which signal]i did the telescopes discover an interference of _ i?’ The second DP un’interferenza (an interference) (the internal argument) is the dependee of the whphrase and neither the subject DP nor the predicate can host this wh- dependency instead.
According to Google Translate (as of 12th July 2019), this second option seems indeed a viable one: (2) What signal did the telescopes find an interference? The translation is ill formed being the internal argument of find filled both by the wh- phrase and the DP an interference (which cannot take a whDP as its own argument due to the absence of a relevant preposition).
In this work we focus on a similar non-local dependency involving two kinds of copular sentences: Inverse (3.a) and Canonical (3.b). Using these constructions, we will test the availability of wh- PP sub-extraction from both the first and the second DP as exemplified in (4). (3) a. le foto del muro sono la causa della rivolta the pictures of the wall are the cause of the riot b. la causa della rivolta sono le foto del muro the cause of the riot are the pictures of-the wall ‘the cause of the riot is the pictures of the wall’ (4) a. [Di quale rivolta]i le foto del muro sono of which riot the pictures of_the wall are la causa _ i ? the cause b. [Di quale muro]i le foto _ i sono of which riot the pictures of the wall are la causa della rivolta? the cause of_the riot In the first part of this paper (§2), we will briefly present an analysis for these constructions, then we will demonstrate that native speakers are selectively sensitive both to the copular structural configuration (Canonical vs. Inverse) and to the extraction site (subject vs. predicate) (§3). In §4 we will test the insensibility of some freely available NLP tools (Google Translate, the Natural Language service of Google Cloud AI API, UDpipe, Stanford Parser and LinguA) to the syntactic oppositions previously discussed.
Copular sentences are those sentences whose
main verb is to be (the copula) and its equivalents
across languages. A subset of copular sentences is
the one involving two DPs, linearly ordered as DP
V DP. Those are dubbed nominal copular
sentences. In this configuration, a nominal phrase
realizes the predicate of the sentence (“the
cause…” in (3)) while the other is the subject of
the predicate (“the pictures…” in (3)). According
to
Traditionally, copular sentences are analyzed
as involving the raising of a DP from the same
base generated structure
These two different representations offer a principled explanation for many asymmetries across languages. Distinguishing between Canonical and Inverse copular sentences is not DPsubj
IP
V DPpred
VP ti IP V
SC
DPpred
VP DPsubj
SC
ti
always easy or possible
the pictures are /*is the cause b. la causa sono/*è le foto
the cause are/*is the pictures (8) a. the pictures are/*is the cause.
b. the cause *are/is the pictures
Extraction is only allowed from the post-verbal DP – the predicate – in Canonical sentences (9), whereas it is not allowed from the post-verbal DP – the subject – in Inverse copular sentences (10). (9) a. which rioti do you think a picture of the wall was the cause of _i? b. di quale rivoltai pensi che una foto del of which rioti do you think that a picture of_the muro sia la causa _i? wall is the cause _i? (10) a. *which walli do you think a cause of the riot was a picture of _i? b. *di quale muroi pensi che la causa della of which walli you think that the cause of_the rivolta sia una foto _i? riot is a picture _i?
Before considering the computational side or the proposed structural analysis we investigated whether the human parser is sensitive to the critical distinctions illustrated here. Two experiments are discussed, testing the processing of Canonical vs Inverse copular sentences (first condition) involving the extraction of a whelement from a DP embedded either under the subject or the predicate (second condition).
Our prediction was that the sensitivity to agreement and to the argumental vs. predicative role distinction for the two DPs involved would have influenced both the online and the offline performance of native speakers: participants should show an advantage in parsing Canonical copular sentences (vs. Inverse ones), since only the Canonical configuration allow the extraction from the predicate DP, whereas all the other kinds of extraction – from the subject in Canonical and from both the subject and the predicate in Inverse – should be disallowed (§2.1).
In order to test these hypotheses, we performed (i) a Self-Paced Reading (SPR) experiment with a Sentence Comprehension Task at the end, and (ii) an Acceptability Judgement Task (AJT). 3.1
In both the SPR and AJT the set of stimuli was the same: 128 items (divided in 4 conditions) and 40 fillers, in SPR, and 60 fillers, in AJT per condition (72 items per experiment in SPR, 92 in AJT). The 2x2 design produced four experimental conditions, exemplified in (11): (11) Condition 1:
*[PP Di quale muro]i … [DP le foto _i]a sono [SC [_a] Of which wall the pictures are [DP la causa [PP della rivolta]]]?
the cause of_the riot?
[PPDi quale rivolta]k … [DP le foto [PP del muro]]a Of which riot the pictures of_the wall sono [SC [ _a] [la causa _k]] are the cause?
*[PP Di quale muro]i…[la causa [PP della rivolta]]b Of which wall the cause of_the riot sono [SC [le foto _i] [ _b]]? are (=is) the pictures?
*[PP Di quale rivolta]k … [la causa _k ]b sono [SC Of which riot … the cause are (=is) [DP le foto [PP del muro]] [ _b]]?
the pictures of_the wall 3.2
32 native Italian speakers participated in the
experiment. Stimuli were composed by questions
and by their answers; participants had to read the
question word by word and, then, the answer.
Finally, they had to judge the appropriateness of
the answer.
Participants showed higher accuracy in answering
to comprehension questions when the extraction
occurred from the post-verbal DP in Canonical
copular sentences – DP predicate in Condition 2
– than in Inverse copular sentences – DP subject
in Condition 3 – while extraction from the Inverse
copular constructions induced lower accuracy
(-0.41, z=‐2.054, p=0.04; Fig. 1). This confirms
that the structural asymmetry between referential
subjects and predicative DPs has a central role in
both the processing and the comprehension of
nominal copular sentences. Similarly, Inverse vs
Canonical opposition seems relevant since
extractions from both sites in the Inverse copular
constructions produce lower accurate answers
compared to the extraction from the predicate in
canonical copulars
40 native Italian speakers participated in the experiment. Stimuli were the same than in SPR. Participants had to rate the acceptability of questions on a scale from 1 to 7. 3.5
The results (fig.2) confirm the previous on-line findings and show that (i) Canonical constructions were more acceptable than Inverse ones and that (ii) among the different types of copular sentences, the ones with an extraction from predicates have higher rates than the ones with extraction from subjects.
To evaluate the state-of-the-art of NLP with
respect to the contrasts we discussed (Canonical
vs Inverse copular sentences) in a configuration
where overt agreement disambiguates the critical
roles (predicate vs subject), we ran few tests using
the following tools:
1. UDpipe
(cloud.google.com) We first tested standard Canonical (3.a) and Inverse (3.b) copular constructions, then we tried to assess qualitatively the output analyses provided by these tools with respect to subextraction from the predicate in Canonical sentences (9.a-b), here repeated for convenience: (3) a. le foto del muro sono la causa della rivolta the pictures of the wall are the cause of the riot b. la causa della rivolta sono le foto del muro the cause of the riot are the pictures of-the wall the cause of the riot is the pictures of the wall (9) a. which rioti do you think a picture of the wall was the cause of _i? b. di quale rivoltai pensi che una foto del muro sia la causa _i? of which rioti do you think that a picture of the wall is the cause _i? 4.1
UDPipe Natural Language Processing - Text
Annotation interface
Stanford parser
LinguA annotation pipeline (service provided
online by ItaliaNLP Lab at Istituto di Linguistica
Computazionale "Antonio Zampolli" ILC in Pisa)
has been used for our tests on Italian,
implementing a version of
We finally investigated the Natural Language service – one of the tools provided by Google Cloud AI Solutions API – which returns syntactic representations of sentences (https://cloud.google.com/natural-language/). While both canonical and inverse copular analyses are equivalent in English to the ones provided by the Stanford Parser (hence partially consistent with our analyses), in Italian, using the Canonical copular sentence ‘le intercettazionik sonok la documentazionei’ (‘the interceptions are the documentation’), the tool incorrectly analyses the predicate DP the documentation as an attribute (fig. 4) (this might be a consistent annotation of all nominal predicates Google adopted, but it is clearly misleading here). Moreover, when it is provided with the Inverse form of the sentence ‘la documentazione sono le intercettazioni’ (lett. the documentation are the interceptions; ‘The documentation is the interceptions’), the tool incorrectly analyzes the raised predicative DP the documentation – singular noun – as the subject, putting it in a wrong agreement relation with the verb (plural form) (Fig. 5). Then, in the end, this parser fails in recognizing the critical difference between Canonical and Inverse copular sentences giving exactly the same analysis for both cases (3.a) and (3.b). In order to evaluate the impact of these wrong analyses on a practical NLP task, we finally carried out our conclusive experiments on one of the most famous and largely exploited machine translation software: Google Translate.
Starting with simple examples, we observed that when the tool is provided with the Italian Inverse copular sentence ‘La causa della rivolta sono le foto del muro’ (lett. the cause of the riot are the pictures of the wall; ‘The cause of the riot is the pictures of the wall’), it gives the wrong
are the photos of the wall’ (Fig.6), in which the verb does not agree with the pre-verbal DP “the cause of the uprising”, contrary to what it does in English (as we saw in 7). Interestingly, reversing the translation from English to Italian the cause of the riot is the pictures of the wall the system correctly produces la causa della rivolta sono le immagini del muro where proper agreement (with the post-verbal subject) is in place. Since the analysis provided by any tool we tested is theoretically inconsistent with this result, we hypothesized that this translation could have been obtained adopting an example-based approach; it was worth then to test if the correct agreement with the post-verbal subject is just an accident (this is a well know prototypical sentence, widely discussed in literature and it might have been included in the Google Translate training set) or if the analysis is generalized of any possible subject/predicate pair.
intercettazioni (lett. the documentation are the interceptions, that means ‘The documentation is the interceptions’) would suit our purpose nicely. In the English > Italian direction the correct singular copular agreement is produced (“the documentation is the interceptions”) but from Italian to English this time the wrong agreement is obtained, totally ignoring the number of the real post-verbal subject (the documentation is the interceptions > la documentazione è le intercettazioni). We concluded then that no deep analysis is attempted so as to distinguish between subject and predicate roles and this turns out to be fatal.
In this paper we demonstrated that nominal copular sentences constitute a clear challenge for the computational analysis since the same string of elements [DP V DP] can have in principle two different syntactic representations (hence two different meanings), depending on which kind of copular sentence is realized (Canonical or Inverse). In this paper, we spotted various glitches in the automatic analyses which in the end led either to significant failures (Google Translate) or to rough structural hypotheses that bluntly ignore the relevant contrasts here discussed. Our empirical study, testing both online and offline the wh- PP sub-extraction possibilities from both subject and predicate DPs, shows that native speakers are sensitive with respect to the different structural roles; in addition, they perceive as expected the underlying structural representation of Canonical vs. Inverse copular construction. None of the NLP tools we tested succeeded in providing a full set of coherent analyses, with the exception of the Stanford Parser for English that at least succeeded in analyzing correctly the canonical copular sentences. This analysis was however insufficient in the case of inverse constructions and in case of sub-extraction, confirming that non-local dependencies are critical configurations native speakers are able to parse but machine do not, yet.