Аннотация This paper describes a pilot study of the problem of detecting singleton mentions in Russian texts. A noun phrase is considered a singleton mention if it is the only referent of some entity. We discuss various morphosyntactic and lexical features, some of which were used for analogous tasks for English and propose new features derived from the discourse analysis. Testing the machine learning classifiers trained with the use of proposed features, we conclude that although the quality of classifiers is significantly lower than for English, they still have rather high precision and thus can be helpful in various tasks of mention tracking.
Coreference resolution is an important preprocessing step for many advanced
NLP tasks that deal with discourse processing. Although this task has been
actively researched for several decades (e.g. [
Coreference resolution task can be seen as a task of linking pairs of mentions
that refer to the same discourse entity. However it has been shown ([
In this paper we conduct a pilot study on singleton detection for Russian. We employ the features that were applied to the similar task for English and check if they work for Russian. In addition we discuss some new, Russian specific features. In the experiment we test different subsets of them and measure the importance of each feature.
The rest of the paper is structured as follows. In section 2 we give an overview of the previous work on singleton detection and some theoretical literature on discourse that deals with certain peculiarities of singleton mentions. In section 3 we describe the corpus that we used for our experiments, the set up for the experiments and their results. In section 4 we analyze the impact of the features and discuss expected and unexpected outcomes. 2 2.1
Among the vast amount of research devoted to coreference resolution for English,
there has been some work done on various aspects of mention tracking, e.g.
discourse-new detection ([18]) finding NPs that start coreferential chains;
distinguishing anaphoric and non-anaphoric NPs ([
The definiteness category is usually an important factor for those tasks, and
some features discussed in the papers rely crucially on the presence of specific
articles (see [
Since Russian is an article-less language, we could not use features based on the overt morphosyntactic definiteness.
The present work is mainly based on Recasens et al. ([
The other cluster of features discussed there is based on information structure, most importantly, syntactic position. Subjects as well as verb arguments have positive association with coreferent use of an NP. 2.2
As for Russian, there is a lack of research concerning the unique or non-coreferring
expressions properties. However, there are works devoted to special devices for
new salient entities introduction into discourse (e.g. [
Among the features used there is NP length (the first NP for a new salient
referent tends to be longer) and, in particular, the number of adjectives. For
instance, [
He came up to the table, took a black pen in the shape of a dagger from it ?and started writing. b. On podoˇsol k stolu, vzjal ruˇcku, sel i stal pisat’.
He came up to the table, took a pen and started writing.
In both examples the underlined entity (the pen) is a singleton. In the example 1a it has two modifiers and the end of the sentence without mentioning the entity again sounds unnatural. In the example 1b the NP is a typical (‘expected’) participant of the ‘writing’ event. Therefore, the sentence seems acceptable.
According to [
Thus, the question is whether those features that showed their usefulness in the first-mentions detection task, could be helpful for the singleton differentiation as well. 3 3.1
Our experiments were conducted on RuCor, a Russian coreference corpus released
as a part of the RU-EVAL campaign ([
The corpus consists of short texts or fragments of texts in a variety of genres:
news, scientific articles, blog posts and fiction. The whole corpus contains about
180 texts and 3 638 coreferential chains with 16 557 noun phrases in total. Each
text in the corpus is tokenized, split into sentences and morphologically tagged
using tools developed by Serge Sharoff ([
Since the RuCor annotation followed MUC guidelines ([
In this pilot study, we tested 4 groups of features: basic, structural, lexical,
and syntactic features. Most of the features that we used were proposed before
for detecting singleton mentions in English (e.g. [
As we mentioned already, our notion of singletons collapses two types of
mentions: those that can not be anaphoric and those that were mentioned only
once in the text. In order to detect both of them, we compiled features that detect
non-anaphoricity with those that should have correlation with the discourse role:
non-coreferent mentions should be less important for the discourse.
Basic features The most basic feature is the number of occurrences of the NP
in question or its head in the text before. It is obvious that if an NP is repeated,
chances are that the entity is not a singleton. Other basic features check whether
the noun phrase is animate, a proper noun, contains non-cyrillic characters or is
a pronoun. Those features were shown to be useful for English in [
Syntactic features Syntactic structure can shed a lot of light on the NPs’
discourse roles. Studies in Centering theory and various discourse studies showed
that coreferent mentions tend to be core verbal arguments and prefer
sentenceinitial positions in a sentence (e.g. [
In order to include this information in the classifier we used the syntactic annotation described in section 3.1. An NP was considered a subject if the type of a syntactic role was ‘предик’. An NP was considered an object if its role was ‘1-компл’4. 4 A submitted version of a paper did not use syntactic information because the experiments showed its low quality. Further experiments showed that although there are a lot of errors in the syntactic information, it still improves the classification quality.
Less standard feature that we employed was if an NP is in genitive case. The source for this feature was the intuition about Russian genitive that it coincides with non-argument positions.
Lexical features We used four precompiled lists to detect non-coreferent mentions: (i) indefinites, (ii) possessive pronouns, (iii) negative pronouns and (iv) non– identity words (‘takoj zhe’ such as, ‘similar’ podobnyj, etc.). If all the previous groups contained features that were designed to detect NPs that are more likely to be unique in the text, judging by their discourse role, those lists (except for the last one) should detect NPs that can not be coreferent. This way we have features for both non-referring and potentially coreferent but unique mentions. 3.3
To test how good various features distinguish singleton mentions from
nonsingleton ones, we have built a set of classifiers using Random Forest classifier
implemented in Scikit-Learn library ([
As it has already been noted, singletons are more frequent than coreferent
mentions. Due to this disproportion the training set is unbalanced there
are more than two times more singleton mentions than non-singleton ones
and this influences resulting classification quality. To overcome this problem, we
performed oversampling on the training set. The best results were achieved using
SMOTE+Tomek method ([
All features 0.600 0.708 0.650 Таблица 1. Classification results (for the minority class)
The basic feature set gives the highest precision, while the recall is lower than using more complex features. The more features are used, the more is recall. Using the full set of features increases the recall by about 12%. 5 An IPython notebook with the experiments is available here: https://git.io/vwE9F.
To understand the individual importance of each feature, we have trained a logistic regression model on the training data. The coefficients for each feature are given in table 2. A singleton is the positive class, so positive coefficients mean a bias towards being a singleton mention.
Estimate
Results for basic features agree with the results reported in [
The NP length (number of modifiers) acts in a less expected way: short and
moderately long NPs tend to be non-singletons whereas only very long NPs tend
to be singletons. However, this agrees with the result in [
The syntactic features also have expected behavior: subject mentions tend to be non-singletons and mentions in genitive are slightly biased towards being singletons. This agrees with previously stated intuition of them being non-topical arguments.
Indefinite (free-choice) pronouns such as lyuboj ‘any’, kazhdyj ‘every’, ktonibud ‘anybody’ tend to be used within unique mentions in the discourse. The most reliable feature for different types of pronouns is the negative pronoun type. This result goes in hand with theoretical assumptions that free-choice pronouns and negative pronouns are non-referential expressions and, thus, they are less probable in in a coreferential chain. 4
The first experiment on the singleton detection has shown that the basic set of features gives relatively high precision and a low recall. Additional features (those taken from previous research for English as well as, relevant for firstmention detection) improve the recall, though there is a small precision loss. As a result, at least 70% of singletons can be filtered out from the mentions set for coreference pairs generation in coreference resolution task which will drastically improve computational efficiency and should improve the resolution quality. 5
In this work we discussed various features used for the singleton detection task as a subtask of coreference resolution systems. We presented a preliminary research on singleton detection with a special emphasis on Russian.
The focus was on the testing features introduced for the corresponding task for English in the previous work, as well as some features used for the discoursenew detection.
We tested classifiers that can distinguish singletons. We also set a baseline for further experiments and tested special lexical features. Though the recall is low we think that the results can be used in the coreference resolution systems to filter out false candidates for the coreferring mentions.
The analysis of the results has shown that some of the lexical features such as special class of pronoun types, for example, negative and free choice pronouns are quite promising for this task and need further investigation and enhancing. The other promising direction is a more detailed investigation of the impact of the grammatical role of a given NP. In our future work we are planning to examine the contribution of more elaborated features.
The authors would like to thank the Lomonosov Moscow University students who participated in the corpus markup, Dmitrij Gorshkov for creating and maintaining corpus annotation software, and Dmitrij Privoznov for his useful comments.
This research was supported by the grant from Russian Foundation for Basic Research Fund (15-07-09306). Список литературы 1. Ariel, M.: Accessing Noun-Phrase Antecedents. Routledge (1990) 18. Uryupina, O.: High-precision identification of discourse new and unique noun phrases. In: ACL Student Workshop. Sapporo (2003) 19. Ward, G., Birner, B.: Information structure and non-canonical syntax. The handbook of pragmatics pp. 153–174 (2004)