The paper introduces Treex CR, a coreference resolution (CR) system not only for Czech. As its name suggests, it has been implemented as an integral part of the Treex NLP framework. The main feature that distinguishes it from other CR systems is that it operates on the tectogrammatical layer, a representation of deep syntax. This feature allows for natural handling of elided expressions, e.g. unexpressed subjects in Czech as well as generally ignored English anaphoric expression - relative pronouns and zeros. The system implements a sequence of mention ranking models specialized at particular types of coreferential expressions (relative, reflexive, personal pronouns etc.). It takes advantage of rich feature set extracted from the data linguistically preprocessed with Treex. We evaluated Treex CR on Czech and English datasets and compared it with other systems as well as with modules used in Treex so far.
Coreference Resolution (CR) is a task of discovering coreference relations in a text. Coreference connects mentions of the same real-world entity. Knowing coreference relations may help in understanding the text better, and thus it can be used in various natural language processing applications including question answering, text summarization, and machine translation.
Most of the works on CR have focused on English. In English, a mention almost always corresponds to a chunk of actual text, i.e. it is expressed on the surface. But Czech, for instance, is a different story. Czech is a typical example of pro-drop languages. In other words, a pronoun in the subject position is usually dropped as it is in the following example: “Honza miluje Márii. Taky <ZERO-on> miluje pivo.” (“John loves Mary. He also loves beer.”) If we ignored Czech subject zeros, we would not be able to extract a lot of information encoded in the text.
But subject zeros are not the only coreferential expression that may be dropped from the surface. Indeed, such zero mentions may appear even in the language where one would not expect them. For instance, the following English sentence does not express the relative pronoun: “John wants the beer <ZERO-that> Mary drinks.”
This paper presents the Treex Coreference Resolver (Treex CR).1 It has been primarily designed with focus on 1It is freely available at https://github.com/ufal/treex resolution in Czech texts. Therefore, Treex CR naturally supports coreference resolution of zero mentions.
The platform that ensures this and that our system
operates on is called tectogrammatical layer, a deep syntax
representation of the text. It has been proposed in the
theory of Prague tectogrammatics [
T-layer is also the place where coreference is represented. A generally used style of representing coreference is by co-indexing continuous chunks of surface text. Tectogrammatics adopts a different style. A coreference link always connects two tectogrammatical nodes that represent mentions’ heads. Unlike the surface style, tectogrammatics does not specify a span of the mention, though. Such representation should be easier for a resolver to handle as the errors introduced by wrong identification of mention boundaries are eliminated. On the other hand, for some mentions it may be unclear what its head is.2
At this point, let us introduce the linguistic terminology that we use in the rest of the paper. Multiple coreferential mentions form a chain. Splitting the chain into pairs of mentions, we can adopt the terminology used for a related phenomena – anaphoric relations. The anaphoric relation connects the mention which depends upon another mention used in the earlier context.3 The later mention is denoted as the anaphor while the earlier mention is called the antecedent.
This work is motivated by cross-lingual studies of coreferential relations. We thus concentrate mostly on pronouns and zeros, which behave differently in distant languages, such as Czech and English.4 Coreference of nominal groups is not in the scope of this work because it is less interesting from this perspective.
However, Treex CR is still supposed to be a standard coreference resolver. We thus compare its performance with three coreference resolvers from the Stanford Core as the module Treex::Scen::Coref in the Treex framework. 2As we demonstrate in Section 5.
3As opposed to cataphoric relations, where the dependence is oriented to the future context.
4A thorough analysis of correspondences between Czech and
English coreferential expressions has been conducted in [
Coreference resolution has experienced evolution typical
for most of the problems in natural language processing.
Starting with rule-based approaches (summarized in [20]),
the period of supervised (summarized in [
The evolution of CR for Czech proceeded in a
similar way. It started during the annotation work on Prague
Dependency Treebank 2.0 [16, PDT 2.0] and a set of
deterministic filters for personal pronouns proposed by [17],
followed by a rule-based system for all coreferential
relations annotated in PDT 2.0 [
Moreover, we rectify a fundamental shortcoming of all
these coreference resolvers for Czech – the experiments
with them were conducted on the manual annotation of
tectogrammatical layer. In this way, the systems could
take advantage of gold syntax or disambiguated genders
and numbers. While the rule-based system [
Treex Coreference Resolver has been developed as an
integral part of the Treex framework for natural language
processing [
Before coreference resolution is carried out, the input text must undergo a thorough analysis producing a tectogrammatical representation of its sentences. Treex CR cannot process a text that has not been analyzed this way. Input data must comply with at least basics of this annotation style. The text should be tokenized and labeled with partof-speech tags in order for the resolver to focus on nouns and pronouns as mention candidates. However, the real power of the system lies in exploiting rich linguistic annotation that can be represented by tectogramatics.
pipelines for Czech and English available in the Treex
framework, previously applied for building the
CzechEnglish parallel treebank CzEng 1.6 [
Sentences are first split into tokens, which is ensured by
rule-based modules. Subsequently, the tokens are enriched
with morphological information including part-of-speech
tag, morphological features as well as lemmas. Whereas
in English, the Morcˇe tool [
A dependency tree is build on top of this annotation,
using MST parser [19] and its adapted version [
The NADA tool [
Transition from a surface dependency tree to the tectogrammatical one consists of the following steps. As tectogrammatical nodes correspond to content words only, function words such as prepositions, auxiliary verbs, particles, punctuation must be hidden. Morpho-syntactic information is transferred to tectogrammatical layer by two channels: (i) morpho-syntactic tags called formemes [13] and (ii) features of deep grammar called grammatemes. All nodes are then subject to semantic role labeling assigning them roles such as Actor and Patient, and linking of verbs to items in Czech valency dictionary [12].
5Russian and German version has been trained on automatic English
coreference labeling projected to these languages through a parallel
corpus. See [
Subject zeros are the most prominent anaphoric zeros in Czech. A subject is generated as a child of a finite verb if it has no children in subject position or in nominative case. Grammatical person, number and gender are inferred from a form of the verb.
Perhaps surprisingly, English uses zeros as well. The coreferential ones can be found in relative clauses (see the example in Section 1) and non-finite verbal constructions, e.g. in participles and infinitives. We seek for such constructions and add a zero child with a semantic role corresponding to the type of the construction. This work extends the original Treex module for English zeros’ generation, which addressed only infinitives. 3.2
Treex CR models coreference in a way to be easily optimized by supervised learning. Particularly, we use logistic regression with stochastic gradient descend optimization implemented in the Vowpal Wabbit toolkit.6 Design of the model employs multiple concepts that have proved to be useful and simple at the same time.
Mention-ranking model. Given an anaphor and a set of antecedent candidates, mention-ranking models [10] are trained to score all the candidates at once. Competition between the candidates is captured in the model. Every antecedent candidate describes solely the actual mention. It does not represent a possible cluster of coreferential mentions built up to the moment.
Antecedent candidates for an anaphor are selected from the context window of a predefined size. This is done only for the nodes satisfying simple morphological criteria (e.g. nouns and pronouns). Both the window size and the filtering criteria can be altered as hyperparameters.
What we denote as an anaphor in the model is, in fact, an anaphor candidate. There is no preprocessing that would filter out non-referential anaphor candidates. Instead, both decisions, i.e. (i) to determine if the anaphor candidate is referential and (ii) to find the antecedent of the anaphor, are performed in a single step. This is ensured by adding a fake “antecedent” candidate representing solely the anaphor candidate itself. By selecting this candidate, the model labels the anaphor candidate as non-referential.
6https://github.com/JohnLangford/vowpal_ wabbit/wiki A cascade of specialized models. Properties of coreferential relations are so diverse that it is worth modeling individual anaphor types rather separately than jointly as shown in [11]. For instance, while personal pronouns may refer to one of the previous sentences, the antecedent of relative and reflexive pronouns always lies in the same sentence. By representing coreference of these expressions separately in multiple specialized models, the abovementioned hyperparameters can be adjusted to suit the particular anaphor type. Processing of these anaphor types may be sorted in a cascade so that the output of one model might be taken into account in the following models. Currently, we do not take advantage of this feature, though. Models are thus independent on each other and can be run in any ordering. 3.3
The preprocessing stage (see Section 3.1) enriches a raw text with substantial amount of linguistic material. Feature extraction stage then uses this material to yield features consumable by the learning method. In addition, Vowpal Wabbit, the learning tool we use, supports grouping features into namespaces. The tool may introduce combinations of features as a Cartesian product of selected namespaces and thus massively extend the space of features. This can be controlled by hyperparameters to Vowpal Wabbit.
Features used in Treex CR can be categorized by their form. The categories differ in the number of input arguments they require. Unary features describe only a single node, either anaphor or antecedent candidate. Such features start with prefixes anaph and cand, respectively. Binary features require both the anaphor and the antecedent candidate for their construction. Specifically, they can be formed by agreement or concatenation of respective unary features, but they can generally describe any relation between the two arguments. Finally, ranking features need all the antecedent candidates along with the anaphor candidate to be yielded. Their purpose is to rank antecedent candidates with respect to a particular relation to an anaphor candidate.
Our features also differ by their content. They can be
divided into three categories: (i) location and distance
features, (ii) (deep) morpho-syntactic features, and (iii)
lexical features. The core of the feature set was formed by
adapting features introduced in [
Location and distance features Positions of anaphor and
an antecedent in a sentence were inspired by [
Many of these features are combined to binary variants by agreement and concatenation. Heuristics used in original Treex modules for some anaphor types gave birth to another pack of binary features. For instance, the feature indicating if a candidate is the subject of the anaphor’s clause should target coreference of reflexive pronouns. Similarly, signaling whether a candidate governs the anaphor’s clause should help with resolution of relative pronouns.
Lexical features Lemmas of the mentions’ heads and their parents are directly used as features. Such features may have an effect only if built from frequent words, though. By using them with an external lexical resource, this data sparsity problem can be reduced.
Firstly, we used a long list of noun-verb collocations
collected by [
Another approach to fight data sparsity is to employ an
ontology. Apart from an actual word, we can include all its
hypernymous concepts from the hierarchy as features. We
exploit WordNet [14] and EuroWordNet [
To target proper nouns, we also extract features from tags assigned by named entity recognizers run during the preprocessing stage. 4
We exploited two treebanks for training and testing
purposes: Prague Dependency Treebank 3.0 [2, PDT] and
sents
words
t-nodes
anaph
Relative
Reflexive
PP3
SzPP3
Zero
Other
Prague Czech-English Dependency Treebank 2.0 Coref
[22, PCEDT] for Czech and English, respectively.
Although PCEDT is a Czech-English parallel treebank, we
used only its English side. Both treebanks are collections
of newspaper and journal articles. In addition, they both
follow the annotation principles of the theory of Prague
tectogrammatics [
Training and evaluation test dataset for Czech are formed by PDT sections train-* and etest, respectively. As for English, these two datasets are collected from PCEDT sections 00-18 and 22-24, respectively.8
In addition, we used the official testset for CoNLL 2012
Shared Task to evaluate English systems [
A basic statistics collected on these datasets is shown in Table 1. The anaphor types treated by Treex CR cover around 50% and 25-30% of all anaphors in English and Czech tectogrammatical treebanks, respectively. The main reason of the disproportion is that we did not include Czech non-subject zeros to the collection (class Zero). Czech subject zeros are merged to a common class with personal and possessive pronouns in 3rd person (class SzPP3), as they are trained in a joint model (see Section 5). Due the same reason, English personal and possessive pronouns in 3rd person form a common class PP3. As the CoNLL 2012 testset has no annotation of relative pronouns and zeros, Treex CR covers 30% of all the anaphors.
8During development of our system, we employed the rest of the treebanks’ data as development test dataset for intermediate testing.
9Reasons for ignoring relative pronouns in OntoNotes are unclear. They might be seen as so tied up with rules of grammar and syntax that annotation of such cases is too unattractive to deal with.
Our system uses two specialized models for relative and reflexive pronouns in both languages. The Czech system in addition contains a joint model for subject zeros, personal and possessive pronouns in 3rd person (denoted as SzPP3). The English system contains two more models: one for personal and possessive pronouns in 3rd person (denoted as PP3) and another one for zeros.
Systems to compare. To show performance of Treex CR
in a context, we evaluated multiple other systems on the
same data. Since currently there is no other publicly
available system for Czech to our knowledge, we compare it
with the original Treex set of modules for coreference. The
set consists of rule-based modules for relative and
reflexive pronouns, and a supervised model for SzPP3 mentions.
It has been previously used for building a Czech-English
parallel treebank CzEng 1.0 [
We also report performance of the English predecessor
of Treex CR used to build CzEng 1.0. It comprises a
rulebased module for relative pronouns and zeros, and a joint
supervised model for reflexives and PP3 mentions. In
addition, we include the Stanford Core NLP toolkit to the
evaluation. It contains three approaches to full-fledged CR
that all claimed to improve over the state of the art at the
time of their release: deterministic [18], statistical [
Stanford Core NLP predicts surface mentions, which is not compatible with the evaluation schema designed for tectogrammatical trees. The surface mentions thus must be transformed to the tectogrammatical style of coreference annotation, i.e. the mention heads must be connected with links. We may use the information on mention heads provided by the Stanford system itself. However, by using this approach results we observed completely contradictory results on different datasets. Manual investigation on a sample of the data revealed that often the Stanford system in fact identified a correct antecedent mention, but selected a head different to the one in the data. Most of these cases, e.g. company names like “McDonald’s Corp.” or “Walt Disney Co.”, have no clear head, though. Therefore, we decided to use the gold tectogrammatical tree to identify the head of the mention labeled by the Stanford system. Even though employing gold information for system’s decision is a bad practice, here it should not affect the result so much and we use it only for the third-party systems, not for our Treex CR.
Evaluation measure. Standard evaluation metrics (e.g.
MUC [
• pred(ai) if the CR system claims ai is anaphoric, • both(ai) if both the system and gold annotation claim ai is anaphoric and the antecedent found by the system belongs to the transitive closure of all mentions coreferential with ai in the gold annotation.
After aggregating these counts over all anaphor candidates, we compute the final Precision, Recall and F-score as follows:
P = ∑ ai pred(ai) both(ai)
R = ∑ ai true(ai) both(ai)
F = 2PR P + R To evaluate only a particular anaphor type, the aggregation runs over all anaphor candidates of the given type.
The presented evaluation schema, however, needs to be adjusted for the CoNLL 2012 dataset. As mentioned in Section 4, in this dataset relative pronouns are not considered coreferential and zeros are missing at all. As a result, a system that marks such expressions as antecedents would be penalized. We thus apply the following patch specifically for the CoNLL 2012 dataset to rectify this issue. If the predicted antecedent is a zero or a relative pronoun, instead of using it directly we follow the predicted coreferential chain until the expression outside of these two categories is met. The found expression is then used to calculate the counts, as described above. If no such expression is found, the direct antecedent is used, even if it is a zero or a relative pronoun.
All the scores presented in the rest of the paper are Fscores.
Results and their analysis. Table 2 shows results of evaluation on the Czech data. The Czech version of Treex CR succeeded in its ambition to replace the modules used in Treex until now. It significantly10 outperformed the baseline for each of the anaphor type, with the overall score by 13 percentage points higher. The jump for relative pronouns was particularly high.
The analysis of improved examples for this category shows that apart from the syntactic principles used in the rule-based module, it also exploits other symptoms of 10Significance has been calculated by bootstrap resampling with a confidence level 95%.
deterministic statistical neural coreference. The most prominent are agreement of the anaphor and the antecedent in gender and number as well as the distance between the two. It also succeeds in identifying non-anaphoric examples, for instance interrogative pronouns, which use the same forms.
Results of evaluation on the English data are highlighted in Table 3. Similarly to the Czech system, the English version of Treex CR outperforms its predecessor in Treex by a large margin of 15 percentage points on the PCEDT Eval testset. Most of it stems from a large improvement on the biggest class of anaphors, zeros. Unlike for Czech relative pronouns, the supervised CR is not the only reason for this leap. It largely results from the extension that we made to the method for adding zero arguments of nonfinite clauses (see Section 3.1). Consequently, the coverage of these nodes compared to their gold annotation rose from 34% to 80%. Comparing these two versions of the Treex system on the CoNLL 2012 testset, we see a different picture. The systems’ performances are more similar, the baseline system for PP3 even slightly outperforms the new Treex CR.
As for the comparison with the Stanford systems, we should not look at the scores aggregated over all the anaphor types under scrutiny, because Stanford systems apparently do not address zeros and relative pronouns.11 In fact, the Stanford systems try to reconstruct coreference as it is annotated in OntoNotes 5.0.
The classes of reflexive and PP3 pronouns are the only
ones within the scope of all the resolvers. The Stanford
deterministic system seems to be consistently outperformed
by all the other approaches. Performance rankings on
reflexive pronouns differ for the two datasets, which is
probably the consequence of low frequency of reflexives in the
datasets. Regarding the PP3 pronouns, Treex CR does not
achieve the performance of the state-of-the-art Stanford
neural system. On the CoNLL 2012 testset it is
outperformed even by the Stanford statistical system.
Neverthe11On the other hand, they address coreference of nominal groups and
pronouns in first and second person. Treex CR does not provide Czech
or English models for these classes, so far. Nevertheless, experimental
projection-based models already exist for German and Russian [
To best of our knowledge, no analysis of how Stanford
systems perform for individual anaphor types has been
published, yet. Interestingly, our result show that even
though the overall performance of the neural system on the
CoNLL 2012 testset is reported to be higher [
We described Treex CR, a coreference resolver not only for Czech. The main feature of the system is that it operates on the tectogrammatical layer, which allows it to address also coreference of zeros. The system uses a supervised model, supported by a very rich set of linguistic features. We presented modules for processing Czech and English and evaluated them on several datasets. For comparison, we conducted the evaluation with the predecessors of Treex CR and three versions of the Stanford system, one of which was a state-of-the-art neural resolver for English. Our system seems to have outperformed the baseline system on Czech. On English, although it could not outperform the best approaches in the Stanford system, its performance is high enough to be used in future experiments. Furthermore, it may be used for resolution of anaphor types that are ignored by most of the coreference resolvers for English, i.e. relative pronouns and zeros.
In the future work, we would like to use Treex CR in cross-lingual coreference resolution, where the system is applied on parallel corpus and thus it may take advantage of both languages.
This project has been funded by the GAUK grant 338915 and the Czech Science Foundation grant GA-16-05394S. This work has been also supported and has been using language resources developed and/or stored and/or distributed by the LINDAT/CLARIN project No. LM2015071 of the Ministry of Education, Youth and Sports of the Czech Republic.