Coreference is a basic means to retain coherence of a text that likely exists in every language. However, languages may differ in how a coreference relation is manifested on the surface. A possible way how to measure the extent and nature of such differences is to build a coreference resolution system that operates on a parallel corpus and extracts information from both language sides of the corpus. In this work, we build such a bilingually informed coreference resolution system and apply it on Czech-English data. We compare its performance with the system that learns only from a single language. Our results show that the cross-lingual approach outperforms the monolingual one. They also suggest that a system for Czech can exploit the additional English information more effectively than the other way round. The work concludes with a detailed analysis that tries to reveal the reasons behind these results.
Cross-lingual techniques are becoming still more and more popular. Even though they do not circumvent the task of Coreference Resolution (CR), the research is mostly limited to cross-lingual projection. Other crosslingual techniques remain a largely unexplored area for this task.
One of the yet neglected cross-lingual techniques is called bilingually informed resolution. It is an approach, in which decisions in a particular task are made based on the information from bilingual parallel data. Parallel texts must be available when a method is trained, but also at test time, that is when a trained model is applied to new data. In real-world scenarios, the availability of parallel data at test time requires the technique to apply a machine translation service to acquire them (MT-based bilingually informed resolution).
Nevertheless, for limited purposes it may pay off to use human-translated parallel data instead (corpus-based bilingually informed resolution). If it outperforms the monolingual approach, it may be used in building automatically annotated parallel corpora. Such corpora with more reliable annotation could be useful for corpus-driven theoretical research.1 Furthermore, it can be also used for automatic processing. For instance, improved resolution on 1In case a cross-lingual origin of the annotation does not matter. big parallel data might be leveraged in a weakly supervised manner to boost the models trained in a monolingual way.
The present work is concerned with corpus-based bilingually informed CR on Czech-English texts. Specifically, it focuses on resolution of pronouns and zeros, as these are the coreferential expressions whose grammatical and functional properties differ considerably across the languages. For instance, whereas in English most of non-living objects are referred to with pronouns in neuter gender (e.g. “ it”, “ its”), genders are distributed more evenly in Czech. Information on Czech genders thus may be useful to filter out English candidates that are highly improbable to be coreferential with the pronoun. By comparison of its performance with a monolingual approach and by thorough analysis of the results, our work aims at discovering the extent and nature of such differences.
The paper is structured as follows. After mentioning related work (Section 2), we introduce a coreference resolver (Section 3), both its monolingual and cross-lingual variants. Section 4 describes the dataset used in experiments in Section 5. Before we conclude, the results of experiments are thoroughly analyzed (Section 6). 2
Building a bilingually informed CR system requires a
parallel corpus with at least the target-language side annotated
with coreference. Even these days very few such corpora
exist, e.g. Prague Czech-English Dependency Treebank
2.0 Coref [
It is thus surprising that the peak of popularity for such
approach was reached around ten years before these
corpora had been published. Harabagiu and Maiorano [
As for more recent works, the authors of [
Experiments of Novák and Žabokrtský [
For coreference resolution we adopt a more developed
version of the resolver utilized in [
Treex CR operates on the tectogrammatical layer. It is
a layer of deep syntax based on the theory of Functional
Generative Description [
2Please refer to [
The tectogrammatical layer is also the place, where coreference relations should be annotated. It is technically represented as a link between two coreferential nodes:3 the anaphor (the referring expression) and the antecedent (the referred expression).
Each input text must be first automatically
preprocessed up to this level of linguistic annotation. The CR
system based on supervised machine learning then takes
advantage of the information available in the annotation.
Pre-processing. The input text must undergo an analysis
producing a tectogrammatical representation of its
sentences before coreference resolution is carried out. We
use pipelines for analysis of Czech and English available
in the Treex framework [
Tectogrammatical trees are created by a transformation
from the surface trees. All function words are made
hidden, morpho-syntactic information is transferred and
semantic roles are assigned to tectogrammatical nodes [
Model design. Treex CR models coreference in a way to be easily optimized by supervised learning. Particularly, we use logistic regression with stochastic gradient descend optimization implemented in the Vowpal Wabbit toolkit.4 Design of the model employs multiple concepts that have proved to be useful and simple at the same time.
Given an anaphor and a set of antecedent candidates,
mention-ranking models [
No anaphor detection stage precedes the coreference resolution. Unless another measure was taken, it would lead to all occurrences of the pronoun “ it” labeled as referential, for instance. Nevertheless, the model determines 3A mention is determined only by its head in tectogrammatics. No mention boundaries are specified. Therefore, it is sufficient for a coreference link to determine only two nodes, the mentions’ head nodes.
4https://github.com/JohnLangford/vowpal_ wabbit/wiki whether the anaphor is referential jointly with selecting its antecedent. This is ensured by adding a dummy candidate representing solely the anaphor itself. By selecting this candidate, the model claims that the anaphor is in fact non-referential.
Diverse properties of various types of coreferential relations (e.g. different referential scopes of personal and relative pronouns) encouraged us to model individual anaphor types separately. A specialized model is build for (1) personal and possessive pronouns in 3rd person (and zero subjects in Czech), (2) reflexive pronouns, (3) relative pronouns, and (4) zeros in non-finite clauses. Treex CR runs them in a sequence.
Features. The pre-processing stage enriches a raw text with a substantial amount of linguistic information. Feature extraction stage then uses this material to yield features consumable by the learning method. Features are always related to at most two nodes – an anaphor candidate and an antecedent candidate.
The features can be divided into three categories.
Firstly, location and distance features indicate positions of
the anaphor and the antecedent candidate in a sentence and
their mutual distance in terms of words, clauses and
sentences. Secondly, a big group of features reflects (deep)
morpho-syntactic aspects of the candidates. It includes the
mention head’s part-of-speech tag and morphological
features (e.g. gender, number, person, case), (deep) syntax
features (e.g. dependency relation, semantic role) as well
as some features exploiting the structure of the syntactic
tree. Many of the features are combined by
concatenation or by agreement, i.e. indicating whether the anaphor’s
value agrees with antecedent’s one. Finally, lexical
features focus on lemmas of the mentions’ heads and their
parents. These are used directly or through the
frequencies collected in a large data of Czech National Corpus [
The extension enables bilingually informed CR. Like the monolingual CR, it addresses coreference in one target language at a time. However, instead of data in single language, it must be fed with parallel data in two languages. Both language sides (Czech and English in this case) of the data must be first pre-processed with the pipelines analyzing the texts up to the diagrammatically layer. Furthermore, to facilitate the access to important information in the other language, the pre-processing stage also seeks for alignment between tectogrammatical nodes. The bilingually informed approach then augments the monolingual features with those accessing the other side of the parallel data. Design of the model remains the same as for the monolingual approach.
Alignment. It is central for our cross-lingual approach to
have the English and Czech texts aligned on the level of
tectogrammatical nodes. The alignment is based on
unsupervised word alignment performed by MGIZA++ [
We extract two sets of cross-lingual features: • aligned_all: it consists of all the features contained in a monolingual set for a given aligned language; • aligned_coref : it consists of a single indicator feature, assigning the true value only if the two aligned nodes belong to the same coreferential entity. This feature can be activated only if there exists a monolingual coreference resolver for the aligned language. We employ Treex CR and its monolingual models for English and Czech, but any CR system, even a rulebased one, could be used.
We do not manually construct features combining both language sides. Nevertheless, such features are formed automatically by the machine-learning tool Vowpal Wabbit. 4
We employ Prague Czech-English Dependency Treebank
2.0 Coref [14, PCEDT 2.0 Coref] to train and test our CR
systems. It is a Czech-English parallel corpus, consisting
of almost 50k sentence pairs (more on its basic statistics is
shown in the upper part of the Table 1). The English part
of the treebank is based on texts from the Wall Street
Journal collected for the Penn Treebank [
Although PCEDT 2.0 Coref has been extensively annotated by humans, we strip almost all manual annotations and replace it by the output of the pre-processing pipeline (see Sections 3.1 and 3.2). The only manually annotated information that we retain are the coreferential links.
We do not split the data into train and test sections. All the experiments are conducted using 10-fold crossvalidation.
Sentences Tokens Tecto. nodes
Personal pron.
Possessive pron.
Refl. poss. pron.
Reflexive pron.
Zero subject Zero in nonfin. cl.
Relative pron.
Other
As mentioned in Section 3.1, our CR system consists
of four models targeting different types of mentions as
anaphors. In evaluation, we split the anaphor candidates to
even finer categories, namely: (1) personal pronouns, (2)
possessive pronouns, (3) reflexive possessive pronouns,
(4) reflexive pronouns, all four types of pronouns in the
3rd or ambiguous person, (5) zero subjects, (6) zeros in
non-finite clauses, and (7) relative pronouns (the statistics
of coreferential mentions is collected in the bottom part of
Table 1). Driven by the findings in an analysis of
CzechEnglish correspondences [
The following experiments compare the performance of the monolingual and bilingually informed system. Both systems are trained on the PCEDT dataset. All the design choices (except for the feature sets) and hyperparameter values are shared by both systems.
Evaluation measure. We expect different mention types to
behave differently in the cross-lingual approach. Standard
evaluation metrics (e.g. MUC [
The results of experiments undoubtedly show the superiority of the cross-lingual CR over the monolingual one. Here, we delve more into the comparison of these two approaches. Firstly, we conduct a quantitative analysis of resolvers’ decisions. It should show how many decision changes the switch to the cross-lingual approach introduces for individual mention types and what is the role of anaphoricity in these changes. Secondly, we inspect randomly sampled examples in a qualitative analysis. We attempt to disclose what are the typical examples when the system benefits from the other language and, on the other hand, if there is a systematic case when the cross-lingual approach hurts. Let us start with a quantitative analysis of improvements and worsenings with respect to anaphoricity and type of the anaphor candidate. Tables 3 and 4 show for Czech and English, respectively, how often the cross-lingual system (denoted as C) is better than the monolingual (denoted as M). Each anaphor candidate falls to one of the four categories based on how C and M decided on the candidate: • both decisions were the same and correct (Both X), • both decisions were the same but incorrect (Both ×), • M’s decision was correct while C’s decision was incorrect (M > C), • C’s decision was correct while M’s decision was incorrect (M < C).
A decision is either assignment of the anaphor candidate to a coreferential entity5 or labeling it as non-anaphoric. The tables also distinguish if the candidate is in fact anaphoric or non-anaphoric. Numbers in the tables represent proportions (in %) of these categories aggregated over all instances. Every row thus sums to 100%.
Conditioning on anaphoricity allows us to directly relate this analysis to the anaphora scores shown in Table 2. Note that while resolution on anaphoric mentions may have an effect on both the precision and the recall component of the anaphora score, resolution on non-anaphoric mentions affects only the precision.
Changed decisions account for around 10% in both Czech and English. More importantly, whereas we see over 7% of decisions changed positively in Czech, it corresponds to 5.5% of decisions in English. This accords with the extents of improvement observed on anaphora score. In Czech, a difference between improved and worsened decisions is only a bit higher for anaphoric mentions. It means that the positive effect of English on resolution 5Some of the anaphors that were assigned to the same entity (columns Both X and Both ×) may have been in fact paired with different antecedents by each of the CR algorithms. As our anaphora score is agnostic to such changes, we do not distinguish such cases. of Czech anaphoric mentions is about on par with its effect on resolution on non-anaphoric mentions. But conversely, Czech helps more in resolution of non-anaphoric mentions.
Let us zoom in to the individual mention types. The highest proportion of changed decisions appears for personal pronouns and zero subjects in Czech (14% instances) and for reflexive pronouns in English (12%). Interestingly, its effect on anaphora score cannot be more different. Czech personal pronouns and zero subjects are the mention types where the cross-lingual approach improves the anaphora score the most. On the other hand, English reflexive pronouns are the only mention type for which the resolution deteriorates with cross-lingual features. The systems’ decisions differ the least for Czech reflexive possessive (7%) and English relative pronouns (6%). Here, we also observe a various effect on anaphora score. While the resolution of Czech reflexive possessives is hardly improved by English features, the small amount of changed decisions on English relative pronouns suffices to achieve one of the biggest improvements among English coreferential expressions.
Anaphora scores in Table 2 have already shown that basic reflexive pronouns are the only mention type, where the cross-lingual approach falls behind the monolingual one. The quantitative analysis of changed decisions confirms it, especially for anaphoric occurrences.
The gains of the Czech cross-lingual system on nonanaphoric mentions can be attributed mostly to zeros. Also thanks to the resolution on non-anaphoric mentions, the highest margin between the proportion of improved and worsened instances (5%) is observed on Czech zero subjects. It leads to one of the biggest improvement in terms of the anaphora F-score (see Table 2). 6.2
In the following, we scrutinize more closely what are the typical cases, where the cross-lingual system makes a different decision.
Let us start with a motivating example. Results in Table 2 show that improvement of the bilingually informed system on Czech personal and possessive pronouns and zero subjects is twice as high than on their English equivalents. This observation genuinely surprised us. We had expected the opposite. Our supposition was based on the fact that Czech grammatical gender is more evenly distributed over nouns. We assumed Czech gender could help filtering out the English antecedent candidates whose Czech counterparts do not match the pronoun’s counterpart. Although this still may be true, obviously, there are even stronger factors that operate in the opposite direction – from English to Czech.
Czech personal and possessive pronouns are the mention types that considerably benefit from the cross-lingual approach. Gender of the corresponding English pronoun appears to play an absolutely decisive role. Many times, gender of the Czech pronoun is masculine or feminine while gender of the English pronoun is neuter, as it is in Example 1. English pronoun’s gender thus serves rather as an animacy feature, which cannot be reconstructed solely from the Czech pronoun. The correct antecedent is sometimes selected also with a help from the English pronoun’s number.
Oponentim.pl soudcem.sg Borkam.sg zvolili bojištˇen.sg opponents of judge Bork chose the battlefield drželi homn.sg held it Oponenti soudce Borka zvolili bojišteˇ, drželi ho a udrželi si ho.
Mr. Bork’s opponents chose the battlefield, held it and kept it.
The analysis also shows that English syntax, which is more strict and thus easier to reconstruct, often helps in determining the correct antecedent. Example 2 shows the case, where neither English gender nor number could affect the resolver’s decision. The correct decision is rather a result of a clear structure, where the syntactic objects in coordinated clauses very likely refer to the same entity. kdo posbíral who collected do iniciativy into an initiative Van de Kamp je ten, kdo posbíral plány r˚uzných radikálních ekologických skupin a sesmolil je do jedné neohrabané iniciativy. . .
plánym.p skupinf.p a sesmolil jemfn.p plans from groups and cobbled them (1) (2) Mr. Van de Kamp is the one who collected the plans from the various radical environmental groups and cobbled them into a single unwieldy initiative. . .
Some of the possessive pronouns benefit from another syntax-related factor. Example 3 shows the case where the correct decision was very likely affected by the fact that the aligned English possessive pronoun (“ its Opel line”) is in a short context preceded by a construction with a possessive adjective (“ GM’s interest”). Not only the possessed objects does not have to be the same, but the possessivity factor also suppresses the unclear gender agreement in Czech (“ jeho /its/” can be of masculine or neuter gender, whereas “ spolecˇnost /company/” is of feminine gender and the gender of “ GM” may be arbitrary). (3) zájemm.s spoleˇcnosti GMfm.s o spolecˇnost Jaguarfm.s interest GM-company’s in Jaguar company odráží touhuf.s pomocif.s zpestrˇit produktym.p reflects a desire to help diversify products této spoleˇcnostif.s na trhum.s s vozym.p . jehomn.s of this company in market with cars . its série Opel line Opel Zájem spolecˇnosti GM o spolecˇnost Jaguar odráží touhu pomoci zpestrˇit produkty této americké spolecˇnosti na rostoucím trhu s luxusními vozy. Jeho série Opel má zavedený image. . .
GM’s interest in Jaguar reflects a desire to help diversify the U.S. company’s products in the growing luxury-car segment of the market. Its Opel line has a solid image. . .
Zero subjects is another Czech mention type for which a large improvement of the cross-lingual approach is observed. Anaphoric zero subjects benefit from the aspects similar to those we mentioned for personal pronouns: gender and number of the anaphor, more strict syntactic constraints in English etc. English gender may be even more important here, as the gender of a subject zero is impossible to be recognized just from the form of the governing verb, if the verb is in present tense.
While inspecting a sample of changed decisions for English personal and possessive pronouns, we do not witness many examples of clear influence by Czech gender or number. As for the personal pronouns, influence of gender or number is most often combined with the pure fact that the English pronoun has an aligned counterpart in Czech. For many of such pronouns, the option that the pronoun is non-anaphoric can then be discarded. The strength of this aspect very likely accounts for the fact that the majority of most confident decision changes were in fact labeled as non-anaphoric by the monolingual system (e.g. in Example 4). Czech language side of the data thus help correctly label these pronouns as anaphoric. (4) Compelled service is unconstitutional It is also Nucená službaf.s je protiústavní ∅f.s Je také unwise nerozumná Compelled service is unconstitutional. It is also unwise and unenforceable.
Nucená služba je protiústavní. Je také nerozumná a nevynutitelná.
Similarly, most of the improvements among English possessive pronouns do not result from additional information on gender and number from Czech. The crosslingual system rather takes advantage of the cases where a reflexive possessive pronoun is a Czech counterpart of the English possessive pronoun (see Example 5), or the cases where the pronoun has no Czech counterpart at all. In all these cases, the syntactic subject of the clause in which the pronoun lies is a preferred antecedent. (5) Digital Equipment Corp. announced its line spolecˇnost Digital Equipment Corp. prˇedstavila svou rˇadu of computers pocˇítacˇ ˚u The hottest rivalry in the computer industry intensified sharply yesterday as Digital Equipment Corp. announced its first line of mainframe computers. . .
Nejžhaveˇjší rivalita v pocˇítacˇovém pr˚umyslu se vcˇera notneˇ prˇiostrˇila, když spolecˇnost Digital Equipment Corp. prˇedstavila svou první rˇadu centrálních pocˇítacˇ ˚u. . .
Back to the Czech zero subjects. Many of these expressions reconstructed during the automatic analysis are in fact superfluous. It is usually a consequence of a parsing error, when the real subject of a clause is not recognized (e.g. the word “ spolecˇnosti /companies/” in Example 6). This error subsequently propagates to a wrong decision of the monolingual resolver (the word “ zpráva /report/” labeled as an antecedent). Any superfluous zero subject may be correctly resolved in two ways: (1) labeling it as nonanaphoric, or (2) linking it to the expression that plays the same role in the sentence. We observe that 85% of the decisions corrected by the cross-lingual system are fixed in the former way. And a missing English counterpart of the superfluous zero plays a significant role in such decisions. (6) Avšak zpráva uvádí že spolecˇnostisubj ∅subj But the report said that companies – platí více daní are paying more taxes Avšak zpráva uvádí, že acˇkoliv spolecˇnosti platí více daní, mnoho jich stále platí méneˇ, než cˇiní zákonná sazba.
But even though companies are paying more taxes, many are still paying less than the statutory rate, the report said.
In a similar way, detection of English non-anaphoric zeros in non-finite clauses can be boosted by Czech features. If the zero is non-anaphoric, its governing clause usually remains non-finite in Czech or it turns into a noun phrase. For instance, in Example 7 the entity which performs the act of “ hiring” is not specified in the context of a given sentence, which is emphasized by the use of the noun “ nábor” as a Czech translation of the participle. The automatically parsed structure of such cases is the same: since Czech non-subject zeros are rarely reconstructed by Treex linguistic pre-processing, there is usually no counterpart for the English zero to align with. (7)
Fear of AIDS hinders ∅actor hiring Strach z AIDS komplikuje – nábornoun Fear of AIDS hinders hiring at few hospitals.
Strach z AIDS komplikuje nábor v neˇkolika nemocnicích.
The category of relative pronouns specified in terms of automatically set attributes may contain lots of pronouns that are in fact interrogative or fused. Such instances account for the majority of non-anaphoric English relative pronouns, correctly discovered by the cross-lingual system but not by the monolingual one.
Finally, we sought for the reasons of worsenings within a category of Czech and English reflexive pronouns. The worst decisions made by the cross-lingual method in Czech are on the pronouns that ended up resolved as non-anaphoric. Most of the time these incorrectly labeled pronouns have no alignment to English, thus no cross-lingual features related to the anaphor can be activated. On the other hand, the English cross-lingual resolver makes the most serious mistakes by selecting a wrong antecedent. In these cases, the pronouns are most often aligned to their Czech counterparts and these counterparts are actually often correct. Yet, the choice of the English antecedent seems to be random, regardless whether the Czech counterpart is labeled as coreferential with its correct antecedent, or the counterpart is any of the words sám or samotný, which should indicate emphatic use of the English reflexive pronoun. 7
This work conducts experiments on bilingually informed coreference resolution on Czech-English data. Comparing this cross-lingual approach to a monolingual resolver, we discovered that English helps more in resolution of Czech expressions than vice versa. A quantitative analysis shows that while English facilitates resolution of both Czech anaphoric and non-anaphoric mentions, Czech primarily helps to identify non-anaphoric mentions. The qualitative analysis reveals main reasons for improvements and worsenings all over the mention types. The most surprising finding is that the information on English gender seems to be improving resolution of Czech coreference more than vice versa. The animacy feature hidden in English gender appeared to be stronger than more even distribution of Czech genders across nouns.
This project has been funded by the Czech Science Foundation grant GA-16-05394S. This work has been also supported and has been using language resources developed and/or stored and/or distributed by the LINDAT/CLARIN project No. LM2015071 of the Ministry of Education, Youth and Sports of the Czech Republic.