Coreference resolution is a fundamental ingredient for many downstream tasks in natural language-based applications. For Russian language, the work in coreference resolution is very limited. In this publication, we present a system inspired by the mention-ranking approach, which improves the state-of-the-art F1 score from 0.63 to 0.71, measured with the B3 metric. We evaluate various sets of feature combinations, and also discuss the limitations of the presented work.
Coreference resolution is an important problem in many natural language processing tasks. It can support, i.e., automatic text summarization, knowledge extraction, objects’ identification in dialogue and translation systems [Toldova, Ionov 2017]. The term coreference denotes the relation between the parts of a text (mentions), that refer to the same real world entities, for example the mention of a person name in a text, and of a pronoun that refers to this same person. Thus, the task of coreference resolution is to find and group all the mentions in the text according to their referents. Mentions are typically represented by noun phrases (NPs), named entities and pronouns, except the cases of abstract anaphora, where the anaphoric pronoun refers to the whole preceding sentence and not to noun phrases (NPs) or pronouns [Nedoluzhko, Lapshinova-Koltunski 2016]. In the pair of mentions the first one (full mention) is called the antecedent while the second one is an anafor. In the broader field of coreference resolution, there are two main tasks: mention extraction and coreference resolution in narrower sense (mention clustering) [Sysoev et al. 2017]. Mention extraction finds textual expressions which are possible elements of coreference chains in unstructured data, whereas coreference resolution groups mentions into clusters, which refer to a single real-world entity. The system presented here focuses on the second task, i.e. coreference resolution in the narrower sense.
For Russian language, the work on coreference resolution is limited. Results reported on the RuCor1 [Toldova et al. 2014] coreference corpus are 60.48 of F1 score for the B3 metric by Toldova and Ionov [Toldova, Ionov 2017], and 63.12 F1 by Sysoev et al. [Sysoev et al. 2017]. The mentioned work applies rule-based and “classical” machinelearning methods like decision trees or logistic regression. In this work, we present an approach using neural networks based on an adapted version of the mention-ranking model by Clark and Manning [Clark, Manning 2016]. With this architecture, we manage to outperform previous work with an achieved F1-score of 0.7131. The B3 metric [Bagga, Baldwin 1998, Amigo et al. 2009] is a clustering metric, which evaluates a gold-standard clustering of mentions against a system-produced clustering.
The paper is structured as follows: After an overview of related work in Section 2, Section 3 introduces the system architecture, and Section 4 discusses the features used, and how features are combined into three different sets. The following section (Section 5) provides the evaluation details for those feature sets and compares the results to the state-of-the-art. Furthermore, difficult cases are discussed. The paper concludes with Section 6. 2
Existing approaches to coreference resolution can be divided into heuristic [Hobbs 1978, Boyarski et al. 2013 ] and based on machine learning (ML) algorithms [Rahman, Ng 2009, Ng 2008, Clark, Manning 2016]. Heuristic methods are built upon a handmade set of rules, which is time- and labour-consuming to construct. On the contrary, ML-based approaches are faster and easier to develop, but they depend on the availability of a coreference dataset of sufficient size and quality to apply supervised learning methods.
Recent advances in coreference resolution for English language include the work of Clark and Manning [Clark, Manning 2016] on a cluster-ranking algorithm that handles entity-level information and eliminates the disadvantages of the mention-pair models. The main benefit of this neural network-based method is that it can distinguish beneficial cluster merges from harmful ones. Central parts of the system architecture used for this publication are inspired by this work.
In general, the state-of-the-art results for Russian language are lower than for English, work on Russian language is rather limited so far [Sysoev et al. 2017]. For Russian language, coreference resolution became a more active research topic with the release of a tagged coreference corpus (RuCor) in 2014 [Toldova et al. 2014]. Toldova and Ionov [Toldova, Ionov 2017] compare rule-based and ML-based methods for coreference resolution using this RuCor dataset, with slightly better performance for the ML-based methods. They reach 31.56 for predicted mentions and 60.48 for gold mentions with the B3 metric for the RuCor corpus. Predicted mentions refers to coreference resolution for mentions which were automatically extracted with a mention extraction module. Sysoev et al. [Sysoev et al. 2017] tackle both mention extraction and coreference resolution. For mention extraction, they use a number of linguistic, structural, etc., features and apply classifiers such as logistic regression, Jaccard Item Set mining and random forest, and
...
Hidden Layer h1 Hidden Layer h2 Hidden Layer h3 Output Layer sm
Additional Features ... ...
ReLU (W1h0 + b1) ReLU (W2h1 + b2) ReLU (W3h2 + b3) Wmh3 + bm sm (a, m) reach an F1 of 63.12 for the gold mentions from the RuCor dataset. In comparison, our approach provides an F1-score about 8 points above previous work. 3
We present a coreference resolution system based on the mention-ranking model by Clark and Manning [Clark, Manning 2016]. The core of the system is a feedforward neural network. Its topology is shown in Figure 1. In a nutshell, the network can be divided into two parts: a mention-pair encoder and a mention-ranking model. The implementation of the system is available on github2. The source code was written in Python, using Keras and Tensorflow for the neural network models.
Antecedent Feature
Vector
Mention Feature
Vector The purpose of the mention-pair encoder is to transform a pair of a mention m and its potential antecedent a into their distributed representations. The mention-pair encoder is implemented as a feedforward neural network with three fully-connected hidden layers of rectified linear units (ReLU): hi(a; m) = max(0; Wihi 1(a; m) + bi) (1) where hi(a; m) is an output of the i-th hidden layer for a pair of mention m and its potential antecedent a. Wi is a weight matrix and bi is the bias for the i-th hidden layer.
The input layer of the mention-pair encoder takes a vector of features of a mention and its potential antecedent as well as additional pair features (all the features are described in Section 4). The output of the last hidden layer is the distributed representation of the pair which is used as an input to the mention-ranking model.
2 https://github.com/annkupriyanova/Coreference-Resolution 3.2
The purpose of the mention-ranking model is to estimate the score of coreference compatibility for the pair of a mention m and its potential antecedent a. To compute this score one applies one fully-connected layer to the distributed representation of the pair rm(a; m),
sm(a; m) = Wmrm(a; m) + bm where sm(a; m) denotes a score of coreference compatibility of a pair of mention m and its potential antecedent a, rm(a; m) is the distributed representation of this pair. 3.3
For pretraining, which determines the initial configuration of the model parameters, we used the following objective function,
N X
X log p(t; mi) +
X log(1
p(t; mi)) i=1 t2T (mi) f2F (mi) where T (mi) and F (mi) are sets of true and false antecedents of a mention mi respectively, and p(a; mi) = sigmoid(s(a; mi)).
The main training objective is a slack-rescaled max-margin which penalizes different types of errors:
N X max i=1 a2A(mi) (a; mi)(1 + sm(a; mi) sm(t^i; mi)) where A(mi) is a set of candidate antecedents of a mention mi, t^i is a highest scoring true antecedent of mention mi:
t^i = arg t2mT (amxi) sm(t; mi) and (a; mi) is a cost function for different types of mistakes: (2) (3) (4) (5) (6) (a; mi) = (
F N F A W L 0 if a = N A ^ T (mi) 6= N A if a 6= N A ^ T (mi) = N A if a 6= N A ^ a 2= T (mi) if a 2 T (mi) where F N stands for False New mistake, F A for False Anaphor, and W L for Wrong Link. 4
For creating the coreference resolution models, we designed three feature sets. We will compare the results for the individual models in the evaluation section. Table 1 shows how the features are partitioned into our feature sets.
The list of features is inspired by previous work on English and Russian coreference resolution. The feature set I is a reduced version of the features used by Clark and Manning [Clark, Manning 2016]. In feature set II we removed some of the word embedding features, and added features of explicit indication of morphological characteristics and POS-tag agreement, which is highly relevant for Russian as a morphologically rich language3. And in feature set III the cosine similarity between the vectors of mentions heads completely replaces the word embeddings.
For each mention, we build a feature vector. A mention consisting of one word, is represented by a single vector (word embedding). If the mention consists of a group of words, is represented by the average of the embedding vectors of each word in the group. We use word embeddings pre-trained with FastText on the Wikipedia corpus4. 5
This section describes the experimental setup, especially the dataset, and provides the results of the evaluations for the three models introduced in Section 4 in comparison with existing work. Finally, we discuss some of the difficulties and limitations observed with the current architecture.
3Morphological annotation, lemmatization and word embedding models were borrowed from[Kovriguina et al. 2017] 4https://github.com/facebookresearch/fastText/blob/master/pretrained-vectors.md 5.1
With regards to the dataset, we used the RuCor dataset to train and test the coreference resolution system. RuCor is the first open corpus with coreference annotations for the Russian language. It comprises short texts and text fragments of different genres: news, fiction, scientific papers, etc. All the texts are tokenized, split into sentences and parsed syntactically and morphologically. In the corpus, mentions are limited to NPs that refer to real-world entities. Thus, abstract and generic NPs as well as bridging relations and coreference relations with a split antecedent are not annotated. The RuCor dataset contains 181 texts with 156637 tokens and 3638 coreference chains. Based on these numbers, in model training we used a 60-20-20 percent split to create the training, validation, and test datasets.
In terms of the settings of the neural network models, we minimized the training objectives with the Adam and RMSProp optimizers. For regularization, we applied dropout with a rate of 0.3 on the output of each hidden layer. 5.2
For the feature sets three models of the neural network were developed. All the models were pretrained. Pretraining is performed for an initial setup of the weights, and uses binary cross-entropy to compute the loss when training only the mention-ranking model. The results of the experiments are shown in Table 2.
Model I Model II Model III
According to Table 2 Model II shows the best results. We suppose that word embeddings and explicit indication of the matches in morphological attributes and POS-tags positively influence the results. In Model III word embeddings are replaced with cosine similarity between the embeddings for the mentions’ head words. It appears that is may not be sufficient for the identification of their semantic similarity. Model I has the lowest score which might be explained by the absence of the explicit indication of matches in features, which, as stated above, lowers the model quality. Moreover, the non-proportional lengths of the vectors for different features, with 300-dimensional vectors for the head word of each mention in a pair and all the words of these mentions, in contrast with only 38-dimensional vector for the other features, might influence the outcome. 5.3
In Table 3 we compare the results of our system with the state-of-the-art open coreference resolution systems for Russian by Tolodova and Ionov [Toldova, Ionov 2017] and Sysoev et al. [Sysoev et al. 2017]. We compare the B3 metric on the gold mentions, i.e. mentions from the RuCor corpus (not for mentions automatically extracted from the text). All our models surpass existing work with regards to F1 score, with model II giving the best results.
The comparison baselines can be briefly described as follows
Here, we outline some of the problems and errors which have been discovered in the analysis of the predictions of the neural network: 1. Some errors are caused by the wrong annotations in the RuCor coreference corpus, esp. wrong lemmas or morphological attributes in the corpus. For example, for the word “дотком” (dotcom) two different lemmas were found – “ доткома” and “доткомом”. 2. Direct speech mistakes: Pronouns "я" (I) and “ты” (you), if used in a dialogue by different speakers, can be coreferential in a certain context. For example, in case of the following dialogue: – “Я сегодня выполнил работу за два дня.” (Today I have done the work for two days.) – “Ты - молодец! ” (You did well!) The pronouns "я" (I) and “ты” (you) have the same referent – the first speaker. However, the neural network makes this kind of mistakes because there is no information about speakers in the dataset. 3. Context mistakes: They arise when the coreference relation gets evident only after the analysis of the mentions context. For example, the coreference relation between the mentions “ Их Сиятельство” (Their Majesty / Highness) and “ женщина в черном капоте” (the woman wearing black dressing gown) is not clear without the analysis of the context parts of the text. Such types of mistakes can be explained with the difficulty of formalizing semantic information. One possible solutions for this problem might be the use of word embeddings for a longer context or even for the whole text. 4. Split anafor mistakes: For example, the mentions “ они” (they) and “Иван Тихонович и Татьяна Финогеновна ” (Ivan Tihonovich and Tatyana Finogenovna) are coreferential. But the network makes the mistake because in the dataset the morphological attributes (gender, number, animacy, etc.) are identified only for the head elements of the mentions. And in the above stated case there are two heads in the second mention and they differ in gender with each other and in number with the head of the first mention. 6
We have presented a coreference resolution system implementing the mention-ranking approach, and experimented with different sets of feature combinations and evaluated their impact on system quality with the B3 metric. Our best model provides an F1 score of 0.7131 on the RuCor dataset, and exceeds existing work by around 8 points. In future work, there are a number of directions to improve model quality: (i) experimenting with other network architectures, (ii) hyperparameter tuning, and (iii) adding more relevant features into the training process, and finally (iv) applying a cluster-ranking approach to capture additional entity-level information. Furthermore, we plan to extend our system with a mention extraction module.
L.Kovriguina acknowledges support from the Russian Fund of Basic Research (RFBR), Grant No. 16-36-60055. Furthermore, the work is supported by the Government of the Russian Federation (Grant 074-U01) through the ITMO Fellowship and Professorship Program.